The Changing of the Guard

Changing-of-the-Guard

 

IBM, Cisco, Intel, Microsoft, Dell, HP, and other large legacy hardware and software companies have something in common these days. A declining revenue stream. Big time.

IBM_logo     cisco      intel_logo

Its not just one or two companies, its most of the big ones. And the results will ultimately effect employees as each quarter passes and they are forced to reckon with Wall Street earnings and reports. The stock market takes no prisoners.

Online newcomers with ‘disruptive’ business models and software are flourishing. Box, Dropbox, Workday, Amazon, Salesforce, Facebook, LinkedIn, all have reported record quarters. Even Apple despite its recently declining stock price is still growing. Just about anything that has to do with mobile phones and tablets has the ‘Midas touch’.  Google Inc. said last Thursday that its revenue grew 31% in the first quarter, while profit rose 16%.

IBM last Thursday reported its revenue dropped.

Software giant Microsoft Corp., once known for rapid sales of PC software, reported that the business that includes its Windows operating system turned in essentially zero growth

Intel Corp., which has struggled to get its chips into mobile devices reported a first-quarter profit drop of 25% on revenue that declined 2.5%.

Oracle Corp., reported a 1% drop in its revenue in its most-recent quarter.

The disparities are the result of technology shifts—the rise of mobile devices and slowing growth in personal computers, conventional software replaced with online versions and cloud outsourcing by corporations. Companies want to rent software and computer systems. The deals are smaller and take less time to implement. Companies want to get out of the construction business—building and rolling out expensive software and hardware systems.

Web-based technology makes it easier for consumers and corporate employees to try new things and makes it harder for older technology suppliers to keep rolling out huge hardware and software deals month in and month out. Hardware, chips and hard drives get faster, smaller and cheaper now every 3-6 months making large purchases by corporations old before the equipment barely gets installed.

Workday Inc which was founded in 2005 and went public in October, reported that revenue for its fourth quarter ended in February rose 89%

Box Inc., founded in 2005 that lets customers store their data online and tap into it from mobile phones and PCs., revenue grew more than 150% in 2012 and it expects another doubling again this year.

“Their biggest challenge is they live in a world of legacy business models,” said Ed Anderson, an analyst with technology research firm Gartner Inc.

Advertisements

Apps, Software and Video Games shortly will go the way of the DVD – they will live in a ‘cloud’.

Bandwidth is the key to the cloud. If you’ve got enough access to it, meaning if you’ve got a fast enough connection, then you don’t need any physical media or software to live in your PC, Mac or for that matter very soon your mobile phone and tablets.

We used to have giant ‘desktop’ computers that had to have HUGE hard drives in order for us to install many applications. For example, Photoshop, Dreamweaver, MS Office, CAD software, etc. all are very large installation packages. Couple this with your collection of MP3’s, photo’s, video’s and documents and most of us ran out of room on a PC that had 50-100 gigs of space for a hard drive.

The obvious to the consumer

Today, as a consumer we see convenient repositories for photo’s, music and videos and documents. Skydrive, GoogleDocs, Dropbox, Box, Amazon Cloud Drive. Now consumers are beginning to understand and use these places to store what they used to store on their home computers. Why? Several key reasons – first, once uploaded to a large mainstream cloud drive (and I mean to the likes of Google, MS or Amazon) your collection of ‘whatever’ is safe. How many of us have dropped or lost a laptop, had a hard drive fail, spilled coffee on our desks and then PC, etc. If you didn’t back it up to an external hard drive you lost it all. Worse yet, I’ve had friends who did and THAT and the hard drive failed shortly thereafter. Years of precious photos (and now videos more than ever thanks for our mobile phones) you can never get back or thousands of MP3’s gone (at $.99 each). Second, consumers now are getting familiar with storing their digital belongings off site and in a cloud. We hear about Amazon’s or Google’s cloud storage drive initiatives more and more everyday. They are fast becoming the new norm. And third – they are not expensive. Certainly not when compared to a 1.5 Terabyte hard drive that can fail without warning.

The not so obvious to us all

What’s not so obvious to consumers is what’s happening in the enterprise business realm. Years ago, you wanted to put up a business domain web site or had a business that required large databases, some required separate servers for clients that are uber security conscious, some needed to have their domain living on a separate server from others (especially the financial and health industries). Others needed production servers, staging servers and then after testing finally deployed an application or web service. Sometimes IT had to physically travel to the colo facility to apply a ‘patch’ to a newly deployed application and hoped that the patch worked as it was supposed to or else everything came to a screeching halt. Businesses lost money, time, and face sometimes. You’d pay Sun, Oracle, Cisco, EMC, etc. millions to deploy servers and DB’s for your environment. You’d spend money on hiring the right technical IT staff to deploy and sync and stitch all of this together. This WAS the norm.

Enterprise today is all moving into a cloud based environment – virtualization is the norm now.

Sun servers were all the rage in the 90’s. But they were VERY expensive. Robust, great customer service, but very costly. Today, you can run a linux box for a fraction of the cost. No more hard drives or servers (blades or otherwise). You can fire up an ‘instance’ and server through AWS in a few minutes. No going into a colo facility. Start-up’s can get to market almost instantaneously and for far less of a cost. You pay for what you use. No more buying a million dollar license for ATG, Vignette or Broadvision and installing 15 discs in a cage. You rent it now. Patches get uploaded by the cloud vendor in a virtual environment and tested before they are deployed to you.

With the rise of this ‘virtualization’, more and more apps or processes now get built into the browser. Java script was written just for this purpose and has allowed for far more sophisticated applications to run in a network environment and now on browsers. Other software will be embedded in browsers as time goes on that will mimic the functionality and hardware on your PC. You can bet on it.

Platform as a Service (PaaS)

Whereas IaaS (infrastructure as a service) providers offer bare compute cycles and SaaS (software as a service) providers offeraccess to such apps as CRM online, PaaS offerings provide turnkey services for developers to get their apps up and running quickly, no infrastructure concerns needed.

Offered as a service, PaaS runs the gamut from development tools to middleware to database software to any “application platform” functionality that developers might require to construct applications. None of these above services come without their problems. But so did everything else before them.

IaaS focuses on managing virtual machines, and the risks are little different than with other cloud types — here, the main risk is rogue or unwarranted commandeering of services. IaaS requires governance and usage monitoring. But with this comes a good degree of convenience and business ROI.

Some of the most popular cloud services running virtually are; Microsoft Windows Azure, Googles App Engine (which offer a nonSQL relational SQL database service), VMware cloud foundry, Force.com ( from salesforce.com), Heroku (also from SF), Amazon Elastic Beanstalk, Engine Ysrd Cloud (for Ruby on Rails enthusiasts), Engine Yard Orchestra (for PHP enthusiasts) and CumuLogic (for Java developers). Consumers never see or hear any of this but use web services that live on these services day in and day out.

What will be obvious to consumers in about 10 years or less

All of this bring me back around to bandwidth and apps. Once we have enough consumers that have access to real fast broadband (100mbps or more down and ideally 200mbps down), then the Apple and Android app store will disappear. Software discs will become obsolete. Video game installation discs – gone. Why, because once you have enough speed, apps can be loaded and accessed wirelessly via the web. The calls to databases, functionality and such can all be received instantly online. Its already happening, slowly. Examples of this in the entertainment space is Ultraviolet, bring your DVD’s to Wal-Mart and upload them to your digital locker – no more disc. Onlive, Livestream, Gaikai all stream video games without the need for a disc, Netflix (you know about them). Consumers are aware of these, but then you’ve also got GoogleDocs and Skydrive for documents and the creation of word and excel docs. We don’t need an install disc anymore.

Last week, it took me 4 days to upload 12,934 MP3’s to my cloud locker at Amazon Music drive. Less time than I ever thought. Available anytime for me to download if need be. That’s nearly $ 13,000 worth of music, stored for as little as $ 20.00 a year.

Mobile apps, software suites, video game discs, movies, music photos and more will still be here but will not physically be in your home forever. It’s inevitable.

Apple’s Half Approach To The ‘Clouds’

This weeks Apple announcement is not quite as cloud centric as you may think. Unlike Googles approach with having a chromebook browser with Linux running underneath and no local storage, Apple is still tethered to the device we use. It’s a world of ‘apps’.

In Google’s view, you do everything using a browser with no local storage or apps. In Apple’s world, while it has taken an elegant approach to its delivery mechanism and user experience bar none, it is still largely delivering a localized environment.

In Google’s world, chromebooks and other devices like these will still need to grapple with the unreliable world of ‘wireless’ connections – or sometimes lack of them and the consumers long time habit and behavior of wanting the content close by them, local.

With Apple’s announcement, they are positioning themselves to take full advantage of the ‘post’ PC world – that is they know that by 2013 (a scant 2 plus years away).

Gartner and others predict mobile phones and THEIR screens will be the No. 1 way we access the Internet to view the web. Here are some more rather startling mobile facts:

*82 percent of consumers have used their mobile phones in a store, 55 percent in a doctor’s office or hospital, 17 percent during a movie at the theater, 14 percent while flying on a plane and 7 percent during church service. Around 17 percent of mobile users have shown a clerk in a store a picture of a product on their mobile phone, saying in effect, “I want this please,” which is a new shopping behavior that is surprisingly being driven by men. 45 percent of users check their mobile devices first thing in the morning, according to InsightExpress.

*Research has determined that mobile advertising is four-to-five times more effective than online advertising, on average…due to various factors, including lack of clutter in mobile, typically one ad per page, and the mobile pages themselves typically do not have a lot of stuff going on—they tend to be very clean. Also, the proportion of the ad on a mobile screen is greater, so it gets more share of eyeballs.

My takeaway from these numbers is that we are steadily becoming a mobile and tablet world, not a PC one.

This is a world the Apple knows better than anyone and using iCloud, it has taken a very good shot at delivering a cloud experience with what really is a local one.  Apple is extending what Apple does best, its core strengths into the cloud. And this is simply the basic integration of Apple’s hardware and software – their elegant OS.  The major difference being it does not yet rely on the browser as the central driving force in the picture (Google’s chrome) rather in Apple’s view what they are giving us an elegant CMS or content delivery system that we manage.  Google is betting on its browser, and they too know its coming to the small screen, therefore, that’s why we are seeing the Android store downloadable app strategy they are pushing out..

Apple which supports its web apps in the App store will have a rude awakening one day as eventually everyone but them will play on a browser using HTML5, but for now Apple’s user experience is by far the best.  A good example of this is when you go to read GoogleNews on your iPhone using Safari and at the bottom of the screen a small box pops up saying ‘ if you want to access Google News, click here to put this app on your device’. If you agree, a small app-like icon gets created on your iPhone using HTML5 just as if you downloaded it through iTunes.

So, Apple IS a cloud player indeed, distributing its OS X online, supporting over the air updates, allowing iTunes to be streamed to any iOS registered device. And iTunes did something that neither Google nor Amazon has done – signed deals with the major music players for their content (video/films excluded for now). This allows us to avoid the time consuming process of uploading our music collection to iCloud (I think I have about 60gigs of files). We can purchase a subscription to Music Match for $24.99 year, and MM will mirror my music collection with the iTunes store – ALL of my music, not just iTunes purchased music. These tracks can now be streamed back to me from the cloud on any MacOS registered device.

However, unlike other pure cloud players, this isn’t a web based operation for all of this. Apple still is enabling core SDK kits (software development kits) for developers to build in access and API’s (application program interfaces) that will let developers integrate their own apps within Apple’s cloud.

To perhaps make this analogy clearer of why it is not a pure based cloud play, look at iTunes. Your music library stays right where it is, with YOU – MM provides software that identifies songs and tracks you have and purchases you made at iTunes against the vast iTunes catalog of music to support MM. All of this not really ‘cloud’ based, but still local.

For us users, the benefit is an elegant, easy intuitive way to sync our content between all of our tablets and mobile devices (Macs included). And this sync does include most other services and docs Apple’s got to offer, calendars, contacts, documents, online storage and photos.  This is far different than Google that has a true cloud offering using GoogleDocs where you store the document and edit in the clouds.  With Apple, you make changes locally and then those changes are synced to the cloud.

This method allows us to be far less vulnerable to the woes of the wireless world or lack of it at times. And, ultimately, it will keep us all purchasing not just apps but what Apple REALLY wants us to buy – newer iPads, newer iPhones and brand new Macs.  Apple is really in the hardware business, unlike Google that wants to drive everyone to the web on inexpensive chromebooks running Linux to see more advertising or Amazon that wants to drive purchases online. It a half hearted approach but it’s a damn elegant one and one that I am particularly enjoying because everything just works!

Enhanced by Zemanta

Apple’s ‘iCloud’ Just Might Be Netflix’s Achilles Heel.

Apple took a long time to get the Internet. Geeks were still installing FTP clients and web browsers for years after Apple belatedly included TCP/IP and PPP to their OS and, when Apple finally did integrate the Internet into Mac OS, it was in a very tacked on kind of way. A browser, an app for making web pages, eventually a few vertical online stores. I think that’s all about to change tomorrow a the WWDC.

The upcoming ‘iCloud’ announcement will vault Apple into the music cloud business, pitted against Amazon and Google (and a few others, but they are the 900lb. gorillas in the room). Apple has been in the business of selling movies and music for a long time now. Far longer than Google and longer than Amazon, at least digitally (no physical plastic CD). Now they will announce ‘iCloud’.

There have been many guessing at what this will look like and include, and I’ll make a few guesses too and I’m sure not all of them will be correct. But its fun nonetheless to postulate. Netflix is unquestionably the king of movie rentals by far. They have the breadth of product, elegance of delivery online and a reasonable cost/subscription plan. Apple is the king of online movie ‘purchases’. Based upon the fact that Apple has been building out a $1B data center in maiden N.C. , it is more than possible that they have infrastructure to support ‘movie’ lockers. That is, you buy a movie and can now store that film remotely in your cloud ‘locker’. This is the one thing that Netflix (at the moment) can’t replicate very easily.

First, it does not have the infrastructure in place (at the least own the facility) even though they host through Amazon’s EC2.  Yes, they can build it out there, but it would be costly.  Second, to my knowledge ownership is a digital right that must be negotiated and exists separately from a pure rental right with the studios. Something that is NOT easy to get from the Hollywood majors – and I know because I’ve been there before several times before. And third, Netflix core business premise is rentals – it has never been the place we turn to purchase a film thereby making it even harder to shift consumer habits that so far lie with an Amazon or iTunes.

This IMHO, could be considered an Achilles heel for Netflix. Not that they couldn’t get here, but perhaps they will get here AFTER Apple does. And first mover advantage is HUGE online and especially in the entertainment space. An storing your movies is altogether another issue – especially once you begin storing your movies in a cloud. They are NOT easy to move (file size is 750megs -1gb or more compared to a typically small 4-5mb mp3 file) nor would you want to. Right now, people are complaining about how you need to upload your MUSIC files to Google or Amazon’s music cloud offering. Imagine what they’d be saying about uploading movies? Again, this is all a guess of mine. Some other thoughts and guesses about tomorrows announcement by Apple MIGHT be:

 

• Your Mac, Windows, or iOS device can sync with all or part of it in the same way that your iOS devices sync with your computer’s iTunes library today because your music library exists in the cloud now.

• Continuous syncing of iOS devices in real time. The implication is never having to plug your iPhone or iPad in to your computer again. You won’t need a computer to sync anymore.

• One login using your Apple account: On any Mac, sign in as a guest using your Apple account credentials and you’ll be brought to the same desktop you get on your personal machine. Files will be downloaded from the cloud (or your home network) on demand, and you’ll have access to all the apps you’ve purchased via the Mac App Store, downloaded and installed on-demand, and removed securely, along with your data, upon log-out.

• Play music on your mac, then with a tap shift the music to your iPhone when you’re on the go. A sizable portion of the playlist will quickly transfer over so there’s no reliance on continued wi-fi access or 3G streaming. A ‘cloud’ benefit.

Lion and iOS 5 will change the playing field for many. It will be interesting to find out exactly how Apple will do this and when tomorrow at WWDC (Worldwide Developer Conference). You can watch it live here on Monday, June 6th at 10am: http://www.macrumorslive.com/.

Enhanced by Zemanta

The End of An Era – Music Companies, ‘cloud’ services and the ISP’s are laughing all the way to the Bank, courtesy of you and me!

Amazon’s Cloud Drive, Google’s BetaMusic, iTunes upcoming ‘cloud’ offering, current subscription based music ‘cloud’ services and music ‘lockers’ ( eMusic, Spotify, Rhapsody, Thumbplay Music, mSpot, MP3Tunes, and others) are all similar in many ways.

There are slight differences in the cost and the amount of storage for free that you get initially. After that, users will find the old fashioned way we now store and playback music might in fact have been the best and most cost efficient after all.

Today, we all have mp3’s or m4p’s (iTunes) stored somewhere on our computers or in an external hard drive or both. We have our iPod and other devices to playback these files. Load up a playlist and take them with you. Soon, the above mentioned services will offer us the ability to ship all or some of our music collection to what effectively is a hard drive outside our house or computer – essentially letting them live ‘over there’ or wherever that service lives, be it Amazon, Google or Apple. Load up a playlist and playback the music just as we do now.

A few things will change however that will drastically alter not how or what we listen to but what it will cost us to listen to what we now playback for free. And the changes are subtle but substantial. And these changes are all designed to generate money, a lot of it, for 3 separate entities; the music cloud service of your choice, the music companies and your local ISP.

What has been an essentially free activity for all of us (creating and playing back music on our device of choice locally), will now very quickly become an expensive one, remotely. The change has been slowly evolving – with the ISP’s like Comcast, Time-Warner and others that supply us leading the way. They have all decided to ‘cap’ and meter our bandwidth usage under various tiered plans. Just like we get our water and electricity usage metered, so will our ‘internet’ usage.

And that’s old news – I’m not telling you anything you have not already heard before. Soon, we will keenly be aware of how much data we will be using monthly. And now, the new music ‘cloud’ offerings will present us with tiered pricing plans to store our music monthly as well. You might have 10 gigs of music (which is NOT a heck of a lot, personally) today that you want to store on Google’s Beta Music Cloud Drive ( they are just being used as one example). For me, I’ve got a ton more than that and I add to that monthly. So initially, I’ll choose a plan for 10 gigs, but I am 100% sure over time, I will eventually double that.

In addition to those charges I want to turn on my ‘cloud’ player and listen to some tunes being played back at my home, through my PC piped into my speakers in the house. Well that used to be free when I loaded up my player locally on my PC. Now with my house being metered, here’s a rough idea of what I could be faced with.

1GB streamed per month = a little more than half an hour of music per day
3GB streamed per month = about 2 hours of music per day
5GB streamed per month = about 3.1 hours of music per day

For music aficionados, that is not a lot of time spent listening to my music. Now mind you, I don’t have to use a cloud service to listen locally – I can continue doing what I do now. But that also means I’ve got to keep a duplicate set of files. And it does not include any bandwidth for any other activities on the Internet during the month I engage in. If you have a iPhone or other device that plays back music, sure you can stream your collection from that same cloud service, but wait, there’s a data cap on your phone too. But wait, there’s more. The new Chrome notebook offers a plan too when you are NOT connected to WiFi – and it’s not cheap:

• Free 100MB per month (what you get with the first two years of ownership under the current plan): 1 hour and 45 minutes of music playback for an entire month
• $10 for an unlimited day pass: listen all day
• $20 for 1GB of data in a given month: a little over half hour of music per day
• $35 for 3GB of data in a given month: nearly two hours of music per day
• $50 for 5GB of data in a given month: a little over three hours of music per day

All of this cost and metering does not include monthly cloud ‘subscription’ costs. Put it all together and you might be looking at some heavy fees every month that you don’t currently pay storing and playing back your music collection locally or playing back on the road through your iPhone, etc.

Now I am a big cloud advocate – there are some big advantages clearly in storing your collection outside of your house. The biggest single advantage I can think of is a disaster – and they DO happen. Replacing a 60gig collection is not only time consuming and expensive but just go and try to remember what was in your collection of say 40,000 songs – good luck! This alone is reason enough to consider storing your collection remotely. Other disadvantages include getting the songs up there to start and you don’t want to move the collection once you are there. Ever try moving 60gigs quickly – there is no quickly. So choose your service very carefully!

While all of these new music services sound great and offer us new and improved ways to listen to our music, I can’t help wondering if one day a few years back the ISP’s and the music industry got together in one big Hotel room and figured this out as a way to get back all of the lost revenue that the ‘Napster’, ‘Kaaza’ and ‘Limewire’ era sucked out of them. Maybe they will get the last laugh after all. Here’s a better one – how would a Netflix for example, replicate a ‘cloud’ locker storage scenario for movies I might purchase? Could it? Just think of THAT cloud storage plan!! Ouch!

13 Movie Online Services is WAY too many. (PPV Part 2)

Netflix vs. Google TV 2.0 PPV (powered by Honeycomb 3.1) vs. YouTube rentals vs. iTunes vs cable PPV vs VUDU vs. Blockbuster OnDemand vs Facebook OnDemand vs BigStar Movies vs CinemaNow OnDemand vs. Alphaline ( Sears/Roxio) vs. Redbox (due 2011) vs. Flixster via Warner Bros. vs anyone else ?

What happens when the airlines have a fare war? You know, you can fly from NY to L.A. for $xx.xx and then the next thing you know, another airline tops that price by $ 20.00? Or gives you a free bag to carry on board? All of a sudden 5 or more airlines have the same special going on. Who do you fly with? Decisions, decisions… It all begins to seem and look the same to you. You get to the same destination, same approximate times, using the same type of transportation, in the air for just about the same money. Who suffers? Ultimately the carriers do.
Meet the carriers. Not the airlines, but the carriers of movies online. I count thirteen (13) of them – eleven (11) of them are live as we speak. All boasting the same movies for the most part for the same prices. All rentable at the same time for about the same amount of time. And I’m not even counting Redbox as an online rentable service…yet. What’s a consumer to do – who do you choose? And why. Do you ‘subscribe’ to a Netflix monthly or do you pick off a film on a one-off basis from another provider. More importantly, how do all of these guys begin to differentiate themselves from each other? How and where do they market themselves? Netflix is clearly the 900lb gorilla today. I guess iTunes is # 2. But beyond them, I can’t really tell who’s in third place. But more importantly, do I really care? Do I need3 or 5 or 7 similar services? On top of all this, I have Verizon’s FIOS cable service at home with thousands of movies to choose from to watch on any given day/hour.

I have licensed movies before from each of the studios and it was no easy task. Number one, its VERY expensive. Figure an upfront fee to be paid to play, maybe between $500k-$1m. That’s just for starters. Then there are the guarantees against each title licensed. Therefore as a provider of online fare, you’ve got to re-coup that fee with a certain number of minimum rentals or turns of the gate so to speak. With nearly 13 services out there plus cable choices, I’m going to take a guess here a few will not make it. Not only must you guarantee upfront cash, you also must explain how you are going to market the studios films, how you will digitally protect them from piracy ( good luck on that one) and how you will separate yourself from the rest of the online movie ‘noise’. All of this and then compete with the new ‘premium’ $30.00 a pop cable TV onDemand offering ( not that I think that’s going to be too successful – it’s the least of these companies problems).
However, the one issue I have with all these services is this: I am unable to save ANYTHING I purchase or rent for viewing later on a rainy day. If I had a ‘digital’ locker – someplace to hold what I spend my money on to see so I can view it later (more than 24hrs later), that might sway me to use that service ALL THE TIME.

The Great Chaos Monkey!

Apr 25, 2011
Working with the Chaos Monkey

Late last year, the Netflix Tech Blog wrote about five lessons they learned moving to Amazon Web Services. AWS is, of course, the preeminent provider of so-called “cloud computing”, so this can essentially be read as key advice for any website considering a move to the cloud. And it’s great advice, too. Here’s the one bit that struck me as most essential:

We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed system to expect and tolerate failure from other systems on which it depends.

If our recommendations system is down, we degrade the quality of our responses to our customers, but we still respond. We’ll show popular titles instead of personalized picks. If our search system is intolerably slow, streaming should still work perfectly fine.

One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances and services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage.

Which, let’s face it, seems like insane advice at first glance. I’m not sure many companies even understand why this would be a good idea, much less have the guts to attempt it. Raise your hand if where you work, someone deployed a daemon or service that randomly kills servers and processes in your server farm.

Now raise your other hand if that person is still employed by your company.

Who in their right mind would willingly choose to work with a Chaos Monkey?

Angry-monkey-family-guy

Sometimes you don’t get a choice; the Chaos Monkey chooses you. At Stack Exchange, we struggled for months with a bizarre problem. Every few days, one of the servers in the Oregon web farm would simply stop responding to all external network requests. No reason, no rationale, and no recovery except for a slow, excruciating shutdown sequence requiring the server to bluescreen before it would reboot.

We spent months — literally months — chasing this problem down. We walked the list of everything we could think of to solve it, and then some:

swapping network ports
replacing network cables
a different switch
multiple versions of the network driver
tweaking OS and driver level network settings
simplifying our network configuration and removing TProxy for more traditional X-FORWARDED-FOR
switching virtualization providers
changing our TCP/IP host model
getting Kernel hotfixes and applying them
involving high-level vendor support teams
some other stuff that I’ve now forgotten because I blacked out from the pain

At one point in this saga our team almost came to blows because we were so frustrated. (Well, as close to “blows” as a remote team can get over Skype, but you know what I mean.) Can you blame us? Every few days, one of our servers — no telling which one — would randomly wink off the network. The Chaos Monkey strikes again!

Even in our time of greatest frustration, I realized that there was a positive side to all this:

Where we had one server performing an essential function, we switched to two.
If we didn’t have a sensible fallback for something, we created one.
We removed dependencies all over the place, paring down to the absolute minimum we required to run.
We implemented workarounds to stay running at all times, even when services we previously considered essential were suddenly no longer available.

Every week that went by, we made our system a tiny bit more redundant, because we had to. Despite the ongoing pain, it became clear that Chaos Monkey was actually doing us a big favor by forcing us to become extremely resilient. Not tomorrow, not someday, not at some indeterminate “we’ll get to it eventually” point in the future, but right now where it hurts.
Now, none of this is new news; our problem is long since solved, and the Netflix Tech Blog article I’m referring to was posted last year. I’ve been meaning to write about it, but I’ve been a little busy. Maybe the timing is prophetic; AWS had a huge multi-day outage last week, which took several major websites down, along with a constellation of smaller sites.

Notably absent from that list of affected AWS sites? Netflix.

When you work with the Chaos Monkey, you quickly learn that everything happens for a reason. Except for those things which happen completely randomly. And that’s why, even though it sounds crazy, the best way to avoid failure is to fail constantly.

Guest Post by Jeff Atwood

Amazon’s EC2 ‘cloud’ outage is just a minor bump in major right road.

By now you’ve heard about Amazon’s EC2 (Elastic Compute Cloud) cloud service failure, or perhaps felt it. If you use Foursquare or read Reddit, use or Quora (among other services or websites) you no doubt felt the impact.

On 4.21 at 1:48am PDT. Quora even had a fun ‘down’ message: “We’d point fingers, but we wouldn’t be where we are today without EC2.” And this YouTube video:

Lew Moorman, chief strategy officer of Rackspace, said it best “It was the computing equivalent of an airplane crash. It is a major episode with widespread damage”. But airline travel, he noted, “is still safer than traveling in a car” — analogous to cloud computing being safer than data centers run by individual companies.

The fact remains, the cloud model is rapidly gaining popularity as a way for companies to outsource computing chores to avoid the costs and headaches of running their own data centers — simply tap in, over the Web, to computer processing and storage without owning the machines or operating software.

Consumers don’t realize that there are a host of sites that base a majority of their ‘up-time’ on cloud services, including Hotmail and Netflix to name just a few. Netflix was not affected by the recent outage because Netflix has taken full advantage of Amazon Web Services’ redundant cloud architecture (which is NOT inexpensive).

Industry analysts said the troubles would prompt many companies to reconsider relying on remote computers beyond their control. And while discussions surrounding that might happen in the next several weeks, in the long-term cloud computing will continue and thrive and evolve into what most industry experts and others already know it to be – a necessary and valued component of doing any kind of business or having any sort of web presence on the Internet. The truth is, every day many more companies around the globe experience ‘outages’ that take their services and sometimes web site down for hours. Added all together, they add up for far more lost time, money and engineering resources that Amazon’s interruption last week.

This round, the companies that were hit hardest by the Amazon interruption were start-ups who are focused on moving fast in pursuit of growth, and who are less likely to pay for extensive backup and recovery services or secondary redundancy in another data center (or Amazon’s redundant cloud architecture).

One of the things that most people are not aware of is that Amazon has an SLA (service level agreement) which is one of the weakest cloud compute SLA of any competing public cloud compute services, even though its uptime is actually very good. Most providers offer 99.99% or better, with many offering 100%, evaluated monthly, with service credit capping at 100% of that monthly bill. Amazon offers 99.95%, evaluated yearly, capping at 10% of that bill, and requires that at least two availability zones within a region be unavailable. Therefore, companies MUST take this into consideration when choosing a vendor as how it relates to what they do on the internet. Taking a secondary, back-up approach can close some of those holes, but it can get mighty expensive. Amazon’s EC2 pricing overall reflects this type of SLA and the ‘human’ support is not included — because of this aspect it can give a 10% to 20% uplift to the price, and it is geared primarily toward the very technically knowledgeable. Amazon is a cloud IaaS-focused (infrastructure-as-a-service) vendor with a very pure vision of highly automated, inexpensive, commodity infrastructure, bought without any commitment to a contract. Amazon is a thought leader; it is extraordinarily innovative, exceptionally agile and very responsive to the market.

That being said, the recent Verizon acquisition of Terremark should put most Tier 1 vendors on their toes including Amazon. Terremark offers colocation, managed hosting (including utility hosting on its Infinistructure platform), developer-centric public cloud IaaS (vCloud Express) and enterprise-class cloud IaaS (Enterprise Cloud). It is a close VMware partner (VMware is one of its investors), and is generally first to market with VMware-based solutions. It is a certified vCloud Datacenter provider. Some of Terremark’s perceived weak spots can and should now be addressed by the merger between the 2 service offerings, in particular the added personnel to better deliver on customer service and satisfaction (stretched thin’ has been the compliant). Now that it has a substantially bigger war chest from its parent Verizon and Verizon’s exceptional network worldwide (remember Uunet), it can take on and adapt more bleeding edge technologies, which it has done in the past, but has not been able to do so most recently.

Combinations like this will likely increase in this space over time as other vendors realize that 2 can be better than one. The devil is always in the details and the trick here is for company cultures to be merged efficiently with a clear and concise plan laid out for both sets of employees. The last thing you need are internal employees to wonder who is going to be replying to the same RFP (request for proposal) to any particular vendor moving forward. Strong, well thought out details by upper management should avoid these pitfalls for the most part, however, it can be pretty tricky to implement.

Long story short – I’d still bet heavily on the long-term success of this business. It’s a smart, cost efficient and labor efficient business model needed for most start-ups, mid-size and Enterprise clients. The days of sending your IT guys into a cage to update the companies software with numerous discs and software patches hoping that it doesn’t disrupt the companies servers should be long gone.