British Airways has resolved its dispute with real estate specialist CBRE over the cause of the data center outage that grounded as many as 672 flights over three days in 2017.

The data center in question was managed by CBRE at the time.

“British Airways and CBRE are pleased to have reached agreement (with no admission as to liability) and continue to work together,” BA told DCD in a statement.

The outage was estimated to cost the airline £58 million ($74.6m) in passenger compensation claims – it’s not clear whether, or how much, CBRE had to pay to reach this outcome.

Nobody’s fault

Airplane
– Paramount Pictures

CBRE is the world’s largest commercial real estate services and investment firm, with more than 800 data centers under management.

In May 2017, on a public holiday weekend, two of these data centers - Boadicea House and Comet House, located near Heathrow - suddenly went dark, leading to around 75,000 passenger cancellations and throwing London airports of Heathrow and Gatwick into disarray.

The outage affected BA’s entire range of operations including bookings, flight check-in, call centers and the mobile app. “We believe the root cause was a power supply issue,” the company’s CEO, Alex Cruz, said at the time. Problems persisted from Saturday until Monday morning.

It was later suggested the responsibility for the outage lay on CBRE, as an outsourced manager of the facility – however, some experts argued that there must have been serious issues with the design of the data centers and their failover strategy, as no single error should have been able to shut down the servers.

18 months later, after failing to agree on the root cause of the incident, the two companies went to court.

The case is now over, but we still don’t know what had actually happened, or who was actually to blame, illustrating why data center outages remain such a difficult subject.

There are industry initiatives that want to make it easier to report on causes of failure without revealing critical business information - like the Data Center Incident Reporting Network (DCiRN), launched in June 2017.

The idea behind the organization is to serve a purpose similar to the Civil Aviation Authority, which funds plane crash investigations.

”The important thing to understand is so many of the failures are recurring failures,” the Network’s co-founder, Ed Ansett of i3 Solutions, previously told DCD. ”I came to the conclusion some time ago that people were not learning from experience.”

Both DCiRN and data center outages in general will be discussed at the upcoming DCD conference and exhibition in New York.