tag:blogger.com,1999:blog-12074752.post1528229753352251967..comments2009-09-02T09:26:30.398+02:00Comments on Peter's Griddle on Club Cloud Computing.com: Engineering large digital infrastructures is not t...pvehttp://www.blogger.com/profile/08966586301980201453noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-12074752.post-21270806095932950572009-09-02T09:26:30.398+02:002009-09-02T09:26:30.398+02:00Oh, and I saw it linked back from the Google blog ...Oh, and I saw it linked back from the Google blog post. Remember who owns Blogger...Khyronhttp://www.blogger.com/profile/08040617292426246605noreply@blogger.comtag:blogger.com,1999:blog-12074752.post-66219448617442401322009-09-02T09:25:59.390+02:002009-09-02T09:25:59.390+02:00Well, I bring it up because it sounds like a proce...Well, I bring it up because it sounds like a process failure, which you can't really beta test for.<br /><br />(Ok, not entirely true, but a lot more difficult.)<br /><br />When I read your question, I was thinking about a side conversation I had with Jonathan Heiliger @ GigaOm's Structure 09 conference back in June. I asked Jonathan about how they test new features, and he basically saidKhyronhttp://www.blogger.com/profile/08040617292426246605noreply@blogger.comtag:blogger.com,1999:blog-12074752.post-24555046104566335672009-09-02T08:47:46.587+02:002009-09-02T08:47:46.587+02:00Yz, you are quick to respond! Where did you pick i...Yz, you are quick to respond! Where did you pick it up!<br /><br />Beta testing is of course impossible on this scale. I meant it ironically. I work with a number of projects that seem to take forever to engineer out a risk that you can only see in production. You might as well jump in.pvehttp://www.blogger.com/profile/08966586301980201453noreply@blogger.comtag:blogger.com,1999:blog-12074752.post-7561895241279462972009-09-02T08:30:32.561+02:002009-09-02T08:30:32.561+02:00Admittedly, I think it a bad idea to perform this ...Admittedly, I think it a bad idea to perform this kind of maintenance in the middle of the day in North America (which I imagine has more GMail users than most other regions, on a traffic basis). But hey, who am I? Besides, if they've done it before successfully, and I imagine they do it a lot, it's only a problem when something goes wrong.Khyronhttp://www.blogger.com/profile/08040617292426246605noreply@blogger.comtag:blogger.com,1999:blog-12074752.post-71202018201032273702009-09-02T08:27:54.894+02:002009-09-02T08:27:54.894+02:00Beta test what? It was an upgrade taking servers o...Beta test what? It was an upgrade taking servers out service. The process is probably well known, well documented and otherwise reliable. There just wasn't headroom and it led to a series of cascading failures, much like the August 2003 problems in the NE United States.<br /><br />Paul Kedrosky has spoken about these tightly coupled systems and cascading failures before too. Commonly, Khyronhttp://www.blogger.com/profile/08040617292426246605noreply@blogger.com