The US government is still struggling to consolidate its data centers, after nearly a decade of shutting down a sprawling network of duplicate, under-utilized data centers, server closets and single racks.

In its latest report on the Federal Technology Acquisition Reform Act (FITARA) and the Data Center Optimization Initiative (DCOI), the Government Accountability Office detailed the status of the consolidation initiative. The 98-page report can be summarized succinctly: Some progress has been made, but there's still a lot of work to do.

Before we begin

First, it is important to understand how the US government defines what makes a 'data center.'

This has been a point of contention within the nation's bureaucracy itself, with several redefinitions of the term changing the number of data centers calculated. Back in 2010, it only meant sites that were 500 square feet or more, and met stringent availability requirements - criteria that covered 2,094 data centers.

Over the following years, that definition was expanded to include almost anything with a server that provides services (whether in a production, test, staging, development, or any other environment). A room containing only print servers, routing equipment, switches, security devices (such as firewalls), or other telecommunication components, was not a data center.

By May 2018, the government said it had 12,062 data centers (as of August 2017) - although it did not create nearly 10,000 data centers in seven years, most were already there, just not covered by the previous definition. Others were there previously, but had not been included by the 24 government agencies subject to FITARA and DCOI, due to poor accounting practices. It is therefore not clear how many data centers the US government has actually added over the past decade.

Of those 12,062 data centers, the definition is then split into two types: tiered, and non-tiered.

A tiered data center has a separate physical space for IT infrastructure, an uninterruptible power supply, a dedicated cooling system or zone, and a backup power generator for a prolonged power outage. Everything else is a non-tiered data center.

All this may change yet again: The Office of Management and Budget's November 2018 draft guidance redefines a data center as a purpose-built, physically separate, dedicated space that meets certain criteria. It aims to focus federal consolidation and optimization efforts on agencies’ larger, tiered data centers and also de-emphasize the consolidation of non-tiered facilities and other smaller spaces.

Close it up

With that out of the way, how well did the 24 agencies perform at closing the data centers, large, small, tiered and non-tiered?

Some progress was made - indeed, of 12,166 data centers, the government has closed some 6,250 as of August 2018, leading to more than $2.37 billion in planned and achieved cost savings and avoidances from fiscal years 2016 through 2018.

The report adds: "In addition, the agencies planned 1,009 closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023 for a total of 1,200 more closures."

This, the GAO believes, is not fast enough. Only 13 of the 24 agencies were on target to meet their closure targets, as set by the Office of Management and Budget (OMB). "Several agencies indicated that they were seeking revised closure goals because they viewed their goals as unattainable. Specifically, officials from Interior’s Office of the CIO stated that a number of the department’s non-tiered data centers were either mission-critical or not cost effective to close."

Two agencies also reported that they did not achieve any fiscal savings from closures. In total, agencies saved $366.5m less than hoped. Billions more could be saved in the future, it is hoped.

The plan is to shut 25 percent of tiered data centers and 60 percent of non-tiered ones by October 1, 2020 (originally, the target was September 2018).

Another aspect of FITARA was to try to improve server efficiency and utilization at the agencies' data centers (in 2009, OMB reported server utilization rates as low as five percent across the federal government’s estimated 150,000 servers).

The OMB targets at least 65 percent server utilization in both tiered and non-tiered data centers.

Just for tiered data centers, it also aims for 100 percent energy metering, 80 percent facility utilization, and a power usage effectiveness (PUE) of 1.5 or lower (1.4 or lower for new builds). It targets a virtualization ratio of four or higher for operating systems to physical servers.

"Agencies reported the greatest progress against two metrics: power usage effectiveness and virtualization metrics. Specifically, 8 agencies reported that they had met OMB’s target for power usage effectiveness and 6 agencies reported that they had met the target for virtualization.

"However, for the energy metering, facility utilization, and server utilization and automated monitoring metrics, no more than 3 agencies reported meeting each."

Five agencies, meanwhile "were not able to report any progress against either or both of the server utilization or power usage effectiveness metrics because their data centers lacked the required monitoring tools to measure progress in these areas."

For those that have consolidated and improved efficiency, some have managed to rely on the data centers they now have to run the majority of workloads. Others have turned to the cloud.

One positive case was in the Department of Commerce agency the National Oceanic and Atmospheric Administration (NOAA). It adopted a cloud-first policy "whenever feasible," which proved vitally useful during the recent spate of hurricanes hitting the US. "NOAA did not have the capacity in its agency-owned facilities to meet the computing demands and requirements of a sudden increase in web traffic on the websites for NOAA and the National Hurricane Center, such as during Hurricanes Irma and Harvey in 2017.

"Commerce stated that using cloud services allowed NOAA to handle 4.7 billion page hits during Hurricane Harvey over a 6-day span, ensuring the websites were not adversely impacted by the increase in traffic."

With all this in mind, the GAO is making 36 recommendations to 22 of the 24 agencies, primarily for the agencies to take action to meet targets.

"Of the 22 agencies to which we made recommendations, 11 agencies agreed with our recommendations; three agencies agreed with some portion, but not all of the recommendations; one agency disagreed with our recommendations; and seven agencies did not state whether they agreed or disagreed with the recommendations."

As with previous reports, the GAO expressed hope that the targets could be met. History paints a picture that is less positive.