Data centers often flaunt their physical security measures, erecting tall fences, blocking entrances with man-traps, and running blanket CCTV coverage to impress the occasional visitor. Servers inside are protected with firewalls and an assortment of software tools, running under the watchful eye of the security operations center.

But there is one aspect of data center security that is rarely talked about, and yet presents a growing problem: the gaping security holes in the mechanical and electrical kit that powers and cools the IT equipment.

The fact that operational technology (OT) systems like chillers and diesel gensets are vulnerable to attacks over network connections is no secret - DCD has reported on the subject repeatedly throughout the years. It ties into the broader debate about the security of industrial control systems (ICS) - the same types of attacks that could be used against data centers could also disable power grids, hospitals, or manufacturing facilities.

Why is the problem still there?

“ICS and SCADA infrastructures are increasingly becoming vulnerable to attack, which can have disastrous consequences for mission-critical infrastructures,” Massimiliano Falcinelli, head of Security Infrastructure at the International Atomic Energy Agency, warned us back in 2016.

At first glance, it should be simple enough to fix – so how come the problem is still there?

Anything you put on a network instantly becomes a target, whether that’s a corporate laptop, a baby monitor, or an air handler. For a simple illustration of this principle, see Shodan – the much-beloved search engine that can track down virtually any Internet-connected device, from sex toys to power plants. The ability to browse random CCTV footage has long been one of its main attractions.

The project was started in 2009 by bioinformatics graduate John Matherly as a way to track the popularity of software tools used on the Internet. The fact people are putting millions of unsecured devices online was discovered almost by accident.

“I wish I could say that I had the foresight that refrigerators would get connected to the Internet, but I didn’t,” Matherly admitted in a recent interview with The Daily Swig. “And I got lucky that people in the security community discovered Shodan.”

Shodan is often used to prove just how many industrial facilities fail at the most basic aspects of security. ‘Industrial control systems’ is a perennially featured category, along with misconfigured databases that are exposed for the whole world to see.

Exploits are available

Once a networked device is discovered, a potential attacker can start researching specific vulnerabilities and exploits to take control, looking to deploy something like the Triton malware, that can trick systems into initiating emergency shutdown.

Gaining access can sometimes be as easy as entering the default password set by the manufacturer. Some of the Internet-accessible ICS used in industrial settings don’t require any authentication at all - having been installed before the Internet went mainstream.

Today, pretty much every piece of data center equipment is networked: this includes CRACs, PDUs, and UPS systems, and the technical space is only going to get more connected.

For example, much-hyped predictive maintenance, in which a machine learning system monitors the state of the equipment and notifies the owner when individual parts appear to be at the end of their lifespan, relies on a tonne of data being exchanged between the piece of equipment and the monitoring system. At the same time, robots are entering the data center, allowing a potential attacker to wreak havoc on a much more granular scale.

Luckily, the list of reasons to physically disable a data center is really short, since conventional cyber criminals are likely to focus their attention on more profitable endeavors. Possible motives include corporate sabotage, hacktivism, and terrorism - and in the current political climate, it is the last option that gets the gears moving.

Closing the holes

In the US, the efforts to close holes in ICS security were spearheaded by the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT). In 2018, the organization became part of a brand new federal agency, the Cybersecurity & Infrastructure Security Agency (CISA), explicitly tasked with protecting the US from “terrorist threats” and armed with a sizeable annual budget of $3 billion in 2020.

Its responsibilities include developing “a comprehensive national plan for securing the key resources and critical infrastructure of the United States, including power production, generation, and distribution systems, information technology and telecommunications systems (including satellites), electronic financial and property record storage and transmission systems, emergency communications systems, and the physical and technological assets that support those systems.”

CISA continues to issue alerts and advisories on ICS attacks and vulnerabilities, but is temporarily in a state of disarray, following the dismissal of its first director, Christopher Krebs, via tweet - ostensibly over suggesting that there was no evidence of any issues with the voting systems used in the recent elections.

In the European Union, a similar role to CISA is fulfilled by the EU Agency for cybersecurity (ENISA) and a network of dedicated Cyber Security Incident Response Teams (CSIRTs) in every member state, established in line with the requirements of the ‘Directive on security of network and information systems’ (NIS Directive) adopted in 2016.

Among other things, the directive asked member states to draw up lists of ‘operators of essential services’ that would be required to pay special attention to cyber security measures, or face penalties – including organizations in energy, transport, banking and finance, healthcare, drinking water distribution, and of special interest to this publication – digital infrastructure. Even though data centers specifically did not fall under the scope of the directive, five EU countries labeled them ‘essential’ anyway.

While Western governments were busy creating new specialist agencies, the security industry responded to the warnings of an impending IoT-pocalypse with a new generation of software tools aimed squarely at ICS systems. Many familiar names went after the slice of the ICS pie, including Check Point, Kaspersky, Tripwire, and Tenable. Schneider Electric partnered with Nozomi Networks for industrial security. Emerson developed a proprietary Power and Water Cybersecurity Suite.

Get skilled up

As the final piece of the puzzle, there are now plenty of opportunities to update professional skills to face ICS threats in the data center, thanks to training courses and certifications that simply didn’t exist five years ago, from organizations like the SANS Institute and even Datacenter Dynamics’ own DCPro training division.

After nearly a decade of talking about the problem of OT equipment security, it remains in the twilight zone. While there’s broad consensus that attacks against critical infrastructure, including data centers, are possible and would have dire consequences, there is a lack of notable examples of carnage brought by weaknesses in ICS – beyond instances like Stuxnet and alleged attacks against Ukrainian power grids.

This means it gets treated as a problem for another day - a bit like the subject of the safety of Soviet nuclear reactors before the Chernobyl disaster. After all, there are other, less complicated ways to disrupt data centers, like Distributed Denial of Service (DDoS) attacks that simply clog up the network with bogus traffic.

The good news is data center operators don’t have to wait for the equivalent of an open-air reactor core fire to bolster their defenses against these new threats. The techniques developed to secure manufacturing equipment can be applied in data centers, and even the basic measures can go a long way.

The National Institute of Standards and Technology recommends using firewalls or unidirectional gateways to stop network traffic from passing directly between the corporate and ICS networks, applying security patches as soon as they are tested in field conditions, blocking all unused ports, assigning user privileges only to people who are authorized to use the systems, and having a robust incident response plan in place in case the equipment is actually disabled.

For a more thorough understanding of vulnerabilities, an audit of mechanical, electrical and plumbing systems might be in order – and several cyber security firms now offer a professional assessment as part of their ICS security services.

Cyber security of OT equipment in a data center is still a relatively recent field, and its many problems are not going to be solved overnight – but the pressure to secure more visible parts of national critical infrastructure in highly uncertain times will undoubtedly have a knock-on effect on how server farms handle their chillers and PDUs.