Equinix's Docklands IBX LD8 experienced a power outage today causing significant network issues.

The problems at the London data center are ongoing at time of writing (12:14pm BST), and appear to have started early in the morning.

L8 reply

"Today Equinix IBX LD8, in the Docklands, London, UK, experienced a power outage. This has impacted customers who are based there. The outage may have also affected customers’ network services," Equinix said in its first public statement at 12:04pm BST - some eight hours after the issue started.

"Equinix engineers have diagnosed the root cause of the issue as a faulty UPS (uninterrupted power supply) system and we are working with our customers to minimize the impact. We regret any inconvenience this has caused.

"Due to this incident we are allowing customers more flexible access to LD8 working within our Covid-19 restrictions including mandatory temperature checks and wearing face coverings. The safety of our employees and our customers is our highest priority."

Among those caught up in the outage was Giganet, which provided more details on its incident report page.

"We lost both A+B feeds to 1 of our 2 Equinix LD8 racks at approximately 4.23am," the company said.

"This follows a UPS failure, which then triggered the fire alarm in the data center according to reports from Equinix. The rack that we have lost power to houses our core Juniper MX router and Cisco LNS. The Juniper MX router is our core device which is needed for everything in LD8 to function, including terminating a number of leased line connections as well as providing connectivity to our vDC platform. All our equipment power suppliers are dual fed with 'diverse' A+B power feeds provided by the data center - however after this incident we suspect that there is a lack of resiliency and will be sure to raise this after the incident is resolved as this is clearly unacceptable to experience a power outage of this gravity."

The sheer size of the outage appears to have caused issues at other data centers, Giganet said. "Although this incident is affecting Equinix LD8, we are seeing signs of intermittent packet loss/increased latency at varying times for leased line & broadband circuits terminating in our other data center - Telehouse North.

"We are speculating, but due to the scale of this outage, and the carriers & suppliers affected, that there could be knock-on impacts around the carrier ecosystem/networks."

The company added: "The London Internet Exchange (LINX), are currently reporting that 150 of their members are affected by this outage, to provide the sense of scale of this outage."

Epsilon, SiPalto, EX Networks, Fast2Host, ICUK.net, and Evoke Telecom were among the other companies taking to social media to apologize to their customers for the outage knocking out their services. BT is also thought to be impacted, but did not respond to requests for comment.

The former Telecity data center last experienced an outage back in 2016, due to an undisclosed problem with the UPS. “This impacted a limited number of customers, however service was restored within minutes,” Equinix UK MD Russell Poole said at the time.

"There is sadly still no estimated fix time which is most frustrating," Giganet said in its most recent update. "Equinix are being continually chased for updates."

Update:

Customers are queuing to get into the data center, but facing issues as the access control system is also offline, Giganet CTO Matthew Skipsey tweeted:

Update 2:

At time of writing, 6:49pm BST, some customers are reporting services returning, but others are still struggling.

Update 3:

At 9:50pm BST Equinix said that "all services have now been resolved."