The perils of hosting in the Cloud

Dear readers

We’ve had a bit of a horrendous week so far. Our eCommerce engine, the software gubbins that runs our online shop lives in the “Cloud”. This bit of “Cloud” is owned by Amazon and is their European Data Centre, based in Dublin Ireland. Apparently it is huge (we’ve never seen it, bit think it is a big shed of some sort). Anyway on Sunday a nearby transformer (not from the movie) got hit by lightening and exploded and caught fire cutting the power supply to both Microsoft’s and Amazon’s Data Centres.

Amazon Web Services

For some reason that is too complicated for me to be even interested in Amazon’s generators didn’t work so all the hundreds of thousands of servers living in the shed switched themselves off in a huff. Apparently servers switching themselves off in a huff makes techie people have sharp intakes of breath.

So it took clever electricity people and Amazon people a 4 to 5 hours to restore power to the Datecentre. Then they started switching on the servers so customers could restart their applications. However some of the servers that switched themselves off in a huff were grumpy and did not want to play (including hours). (The technical term for this is “corrupt”). Amazon told our eCommerce partner (a different company) to leave it with them and they’d try to jolly the servers along.

Over 24 hours later Amazon contacted our eCommerce partner and told them the servers couldn’t be jollied along and needed the equivalent of a visit to The Priory for a full detox and series of spa sessions to restore them to full vigor. (I’m paraphrasing madly here because the techie explanation is just too exciting).

Upon arrival at the Spa all the servers were booked in and ready for their treatments. It then transpired that the all the treatment lotions were rancid. They were rancid because the spa had mistakenly started adding extra ingredients to the lotions and so they couldn’t be used. Now these lotions are uniquely formulated for each customer (server) and absolutely essential for the successful treatment of general dishevelment and grumpiness in servers.

There was only one thing to be done. New batches of lotion would have to be made by hand from a fiddly set of ingredients. This process takes a very long time. Days in fact. It nearly 2 days to make the batches of lotion and treat the grumpy servers, before they could be pressed back into service.

The effect on us has been traumatic. We’ve consumed quantities of cake, tea and alcohol, even a bottle of Grandmothers Sherry has come under threat of consumption.

So what’s the lesson? Well for us it is still that the Cloud is good, it is the inevitable future for provision of technical services. When it’s on song it is excellent. However the caveat is that support systems and emergency plans need to be so much better because the consequences of a failure on this scale are enormous and correspondingly very difficult and time consuming to recover from.

Ironically Amazon experienced a similar failure in the US in April, clearly they didn’t learn anything from it. So come on Mr Jeff Bezos, take your head out of your kindle ebook thingy, get your DataCentre emergency systems sorted out because businesses like ours that depend on the service you provide and need it to work reliably, plain and simple. You are no longer just selling books, you are playing with the economy and peoples’ livelihoods.

This does not take away from the efforts of the people on the ground at Amazon and at our eCommerce partner who worked tirelessly over the last 3 days to restore service. They kept us informed and while that is important and useful it does hide the fact that 3 days of “no existence” for any business is not good business.

Our site http://www.vidamoda.co.uk providing designer fashion for plus size ladies is live once again don’t forget our 50% off summer sale is still running so come along and have a browse!

Thanks for your patience and we are really sorry our store has not been available to you over the last 3 days.

Joanna Davis

Bookmark and Share

Tags: , , ,

Wordpress SEO Plugin by SEOPressor