Amazon EC2 Goes Down: Start the Cloud Fears

Amazon Web ServicesThe big storms yesterday in the eastern part of the country took down part of Amazon EC2 cloud services in Virginia due to the power outages. The Amazon EC2 outage of course had a cascading effect on customers including some high profile names like Netflix, Pinterest and Instagram. Fortunately, Amazon EC2 recovered quickly and everyone was back up in about 4½ hours.

Amazon EC2 Netflix Outage Tweet

Since I am sure a few colleges and universities were also affected by the Amazon EC2 outage, I imagine the higher ed forums will light up with Cloud FUD (Fear, Uncertainty & Doubt).

Even if the impacts to colleges were incidental. Friday’s outage will still be used in arguments against further use of Amazon Web Services, cloud services in general or other forms of outsourcing to the cloud. Especially because Amazon EC2 had a bit a of a hick-up just two weeks ago.

Don’t misunderstand this. It is important to continue the discussions on improving the reliability of cloud services and building redundancy into an enterprise cloud architecture. It is also to important that we devote time to discussing risk mitigation strategies for cloud services.

Just don’t light your hair on fire or start channeling Chicken Little.

But let us also not lose site of the importance of also continuing the discussions on the campus power outages that also occurred. Let’s be sure we devote a commensurate, and proportionate, amount of energy and thought to the cascading events from losing commercial power and disruptions to on-premise technology based services remembering of course we have seen this drill before and should have it mastered.

OU Lancaster Campus Close

There are of course data center outages from failed transfers to back-up then emergency power and perhaps failures from prolonged operations of emergency power systems or overload conditions from poor capacity planning.

But there are also failures outside the data centers in emergency notification systems and processes reliant on third party providers and carriers. And so on down the line for wireless access, IDF’s and departmental computer rooms even remote applications.

For some campuses still suffering from a loss of commercial power like many in Ohio, their test may just be beginning as they face a protracted outage. This presents an unfortunate opportunity to see if anyone applied the lessons learned from the devastation in the I-95 corridor from Tropical Storm Lee last September or not.

If there is a silver lining it is that unlike Pinterest or Netflix, campuses have the luxury of shutting down and cancelling events over the weekend or until commercial power is restored.

This entry was posted in Cloud Computing, IT Risk Management and tagged , , , , , . Bookmark the permalink.

5 Responses to Amazon EC2 Goes Down: Start the Cloud Fears

  1. Laura says:

    Even though we had power on campus, our community college was shut down upon suggestion from the governor so that the power crews could get to trees more easily and restore power more quickly. That’s another consideration: staying out of the way of the workers. Ohio may have been hard-hit, but West Virginia was completely slammed. What I find interesting is that often LMS and/or CMS are touted as a way to carry on when students cannot get to classes because of a natural disaster, but in this case most students and professors lost power longer than the campus did.

  2. The Higher Ed CIO says:

    Those are great points Laura. If I can elaborate…

    When developing BCP’s and DR plans it is common to focus on the local/isolated event scenario, perhaps because those are the exciting and easier ones to plan for. But when you look at the truly disruptive scenarios many of them are regional events not only affecting your site and its operations, but also the lives of your employees, their families, your customers, and those of everyone your plan depends on. I cover this in an earlier post “Natural Disaster and Hazard Risk Assessment Resources“.

    I hope everything on campus and personally is getting back to normal.

  3. Laura says:

    Thank you for the excellent resource. I’ll have to spend some time looking at that. Campus damage was minimal, and nothing that a few extensions and understanding faculty couldn’t resolve. Everyone has power…I think the longest was 10 days without (mine was 7), and all finally have cable and telephone (three weeks was the longest outage I knew of). As there are many railroad tracks and chemical plants here, it’s not a bad idea to be familiar with a variety of emergency plans. Due to snow, I already know 3+ routes to work. :-)

  4. Pingback: Some notes on using Amazon EC2 « Details on Tech Stuff

  5. Pingback: GoDaddy DNS Outage

Comments are closed.