Lessons learned from the OVH datacenter destruction

Advice

ovh-datacenter-destruction-lessons-itd-clickonsite

A week ago, the impossible, the unimaginable just happened. The OVH Data Center in Strasbourg burned down!
Fortunately, no one was injured but, for sure, some businesses will not recover, and OVH will suffer the consequences for years.
Indeed, we believe that this event will influence further OVH and every NSPs on the planet. For sure it has changed the way we see things at ITD and taught us some lessons.

Before a crisis

  1. Do not collocate your backups with your app. If you do, you expose yourself to a local/regional event and might lose it all.
  2. Consider mirroring. If you do not mirror your app, do not expect a miracle. Your provider will not be reachable for obvious reasons: he is dealing with the crash of his lives and everyone like you wants to solve his own problem. Also, it takes time to recover a backup and even more to rebuild an infra. Before last week, both our client and our team thought that mirroring was a luxury for non-mission-critical providers. We shall now reconsider: Is it acceptable for an organization with hundreds of users to miss the software that orchestrates its operations for up to three days?
  3. A proper Disaster Recovery Plan is a must-have! However, it is not enough, keep things in order.
    “We are only as strong as our weakest link”. The parameters you forgot to indicate in your recovery plan will cost you stress and time…36h, in our case ?.

During a crisis

  1. Take it seriously as soon as you know. In this case, we had five team members ready to react at 7 a.m. that morning, which left us two hours to plan and get organized before the crowd and daily routine showed up.
  2. Also inform your clients as soon as you know, if possible before they feel the pain.
  3. Plan effective protocols for your team and keep your clients informed; in our case, we set meetings every two hours followed by communication towards our clients.
  4. As a boss or a manager, free your agenda, become available. Your team will need you to make decisions and do not expect any other tasks to be worked on from your team other than to face the situation. Needless to say, asking for some news outside the rituals would not help… so don’t.

Last but not least

  1. Dealing with such a situation requires having the right guys on board, committed to their job, to your brand, and to what the clients expect from you.
  2. If you serve Clients as well as you can, if you care for them through the year, they will understand the situation, share your pain, and will even bring their contribution/expertise when a disaster knocks on your door.

Conclusion

“At the end of the day, in terms of service, you get what you are paying for, in terms of people you get what you deserve.”

Jerome PERRET, CEO of ITD.