The disastrous fire, which destroyed OVHcloud's SBG2 data center in Strasbourg on March 10, 2021, may have been caused by moisture around inverters in a power room, according to an accident report published by the French accident investigator BEA-RI.

The report by the official French accident investigator BEA-RI, does not give a definitive cause of the fire, but it confirms that it started in the power room and includes CCTV images of the moment the fire started. Along with plans of the site, details of the outbreak and recommendations on how to avoid similar incidents, the report reveals one fascinating detail: there were high moisture readings near one of the inverters in a power room during the hour before the fire started.

OVHcloud previously told DCD it is unable to comment on the cause of the fire until official publication reports are published. The Bas Rhin firefighters published their report in March, and the BEA-RI report has emerged this week shortly after a class action claiming more than €10 million in damages was filed on behalf of OVHcloud customers.

Drips of information?

The BEA-RI report includes a lot of background on the operation and design of data centers, along with a detailed account of the OVHcloud fire, which started at 12.35am on March 10, simultaneously in two energy rooms.

After the alarm went off, at 12:35am, a guard quickly noted thick black smoke at power room 2 on the ground floor, and evacuated the building by 12:42am. First responders arrived at the scene at 12:59am, but were unable to prevent the fire from spreading to the rest of the building by 01:30am.

The BEA-RI report says "The emergency energies of the buildings are cut at 01:13 (for SBG2) and 01:28 (for SBG3, SBG1 and SBG4) by OVH teams."

However, it wasn't that simple. As the Bas Rhin firefighters' earlier reported, there was still power at the site. The BEA-RI report says the site power supply "is cut at the upstream source substation" by the electricity supplier Strasbourg Électricité Réseaux (RES) at 01:30. But "at 02:14, there is still power in building 2, now totally ablaze, and a significant propagation towards building 1 is observed."

According to the Bas-Rhin firefighters, current was still flowing in some systems till later.

The BEA-RI report suggests that diesel generators were set up solely to keep a continuous power supply, and "start[ed] up automatically despite the ongoing fire," so OVHcloud had to neutralize the emergency generators, so they didn't start up when RES cut the power.

The fire was finally brought under control at 06:45am and extinguished at 10:02am, with firefighters leaving the scene some eight hours later.

Short circuit?

Inside the building. BEA-RI reports that fires started almost simultaneously around 12:35am, in two power rooms while the data center is in a normal operating mode. The power equipment had been regularly maintained, but inverters in power room 2 had needed attention because of "recurring problems switching to bypass mode" which were "unexplained."

Sensors on the inverters showed abnormal moisture levels around 11:15pm and 12:30am, before the fire is shown by video surveillance footage and readings on the UPS monitoring panel.

"The presence of liquid or moisture in an electrical device can cause the formation of a short circuit internal likely to cause the observed damage. However, these elements alone do not... determine the cause of the failure," say the investigators, noting that they were not able to "establish whether it was a measurement error or a humidity peak linked for example to a presence of liquid."

The report says: "It is not possible, at this stage, to establish the cause of the failure at the level of the UPS which could be explained by different hypotheses (presence of liquid or humidity linked to the presence of the nearby, malfunction linked to the maintenance operation carried out on the same morning, operation of the inverter outside normal operating ranges, etc.).

It adds that a final verdict could be reached, however: "Examinations of equipment similar to those in question should make it possible to provide information on the origin of the fire outbreak. The BEA-RI does not rule out at this stage the possibility of completing this report based on the conclusions of the ongoing expertise."

Batteries don't seem to be a concern, however. The report notes that OVHcloud used lead-acid batteries which are not a fire risk compared with lithium-ion, and OVHcloud maintained and replaced batteries according to the manufacturer's recommendations, and the batteries were in a different room.

"We will also note that the battery technology used by OVH (lead battery) is a more robust and stable technology than the lithium solution. Internal short circuits on this type of technology remain rare," the report says.

Spreading the fire

The report notes that fire detectors went off on all six floors of the SBG2 went off very quickly: "It can be seen that during the 15 minutes following the start of the fire, several fire detectors were triggered on the ground floor but also on all floors of the SBG2 building. Some detectors located on the upper floors even trigger minutes after the fire started."

This shows that the building, designed to facilitate air movement to cool the servers, created "a great permeability of the building to the outside air and therefore to fire smoke which can spread quickly and without difficulty."

"In addition to the choice to promote the circulation of air within its facilities by means of openings in facade, the materials chosen to build SBG2 and the absence of overrun floors on the facade did not sufficiently slow down the progression of the fire with regard to the time required to massively commit the water resources needed to bring the fire under control and then extinguish it."

The report confirms that the building had wooden floors rated to hold fire back for one hour, and also lacked an automatic extinguishing system: "OVH has chosen not to equip any of the five buildings in its Strasbourg data center with an automatic fire protection."

One positive noted by BEA-RI is that despite the lack of a fire extinguishing system the fire detectors worked perfectly, ensuring a swift evacuation.

Whose fault is it?

The question of fault remains in the air, as OVHcloud has yet to respond to the class action bought by law firm Ziegler & Associés on behalf of more than 140 customers, which alleges that the company did not do enough to prevent the fire, did not protect customer data well enough from possible loss, and did not offer enough compensation to customers after the incident occurred.