Fires cause a serious financial meltdown to businesses. The losses are even bigger than the hacks because fire incidents can destroy the files on the server forever and hamper communication with other servers. The reputation damage in both cases is equal because clients lose confidence in your safety and protection measures.
In a recent incident, a fire broke in one of the datacenters of OVH in Strasbourg and destroyed one of the datacenters, and set two other data centers offline. One hundred firefighters and 43 fire trucks fought with fire and put an end to it. The damages are big enough for the largest hosting provider in Europe and the third-largest hosting provider in the world. But, thankfully, there are no human casualties, and no one got injured.
Steps OVH took to restore the services back to normal
- OVH is sending emails to the customers, updating them about the status of the situation.
- Running ongoing diagnosis
- Building repair work is being carried.
- Long-distance fiber connections are operational.
- Setup of the network room, Tests for high voltage power supply, and electrical work is scheduled to reactivate by March 15.
- All the other services will be reactivated one after the other.
- Customers who were hosted on the affected datacenters are being given the replacement datacenters in Roubaix and Gravelines. 2100 new servers were delivered on Wednesday, March 10, and 630 new servers were delivered on Thursday, March 11. We are aiming to deliver 15,000 new servers per week.
- Implementation of all DRP (Disaster Recovery Plan) mechanisms with the customers.
- DRP – Disaster Recovery Plan is being implemented with our customers.
- Restoring (Strasbourg) SBG1, SBG3, and SBG4 services.
Here’s OVH’s Plan of Action in Detail
OVH’s CEO Octave Klaba is constantly giving new updates about the situation to keep the customers in the loop. He might as well be tweeting as you are reading this.
Not just OVH, but the businesses and websites hosted on the affected servers are also tweeting regularly to keep their respective clients informed and updated. Team members, the support staff, is working round the clock to restore the systems.
The webmasters who did not back up their websites will be facing difficulty. However, those who took the backup do have a chance to change their DNS (Domain Name System) and point that to another server to upload their website and restore the lost data in a few days.
The incident has impacted websites all over the world. The victims include videogame maker Rust, Lichess.org, telecom company AFR-IX, encryption utility VeraCrypt, cryptocurrency exchange Deribit’s blog and docs sites, the art building complex Centre Pompidou, news outlet eeNews Europe among others. Such disasters remind us of the cons of having our data stored on the cloud. But, honestly, fire can break anytime, anywhere if a proper fire protection system is not installed.
As per OVH, it is speculated that a faulty UPS power supply might have caused the fire. CEO Octave Klaba pointed that the images from the thermal cameras suggest the two UPS power supplies were the cause of this incident. More details will be coming soon in the coming days.
Infrastructures with good fire protection systems can contain the fire in time to control the damages. The scale and damage caused by a fire at data centers are enormous because multiple businesses are impacted.
Recent Update: “We plan to restart SBG1+SBG4+the network by Monday, March 15 and SBG3 by Friday, March 19.” – Octave Klaba on Twitter.
I will be posting updates about this incident here, so stay tuned to this space.
