If you were to peek inside the average data center, most enterprises have one thing in common - the infrastructure manager probably maintains about 5kW per rack capacity in 2015. So, how is that going to change?
Most data centers will hit a capacity ceiling in the next five years, and approximately a third (36 percent) will hit that high mark in less than two years, according to the Fall 2014 Data Center User’s Group Survey conducted by Emerson Network Power. A lot of that computational projection comes from the current rack capacity, so how many kilowatts per rack will data centers be using in the next five years?
Resistance to change
“As is often the case, there isn’t a simple answer,” says Richard Villars, vice president of data center and cloud at Cambridge, Massachusetts-based International Data Corporation (IDC).
Villars isn’t being evasive. It turns out, experts can’t really agree, not even on the kind of operations running averages of more than 5kW per rack in their data center today, much less in five years. In short, it depends on whether you’re grounded in a tight budget or have the robust infrastructure demands akin to that of a cloud provider.
“The power density of new and existing data centers varies widely depending on location, size, workload, operational styles of the owners, and the experience and skills of the designers, but there are some common factors and general patterns that are useful,” says Richard Fichera, vice president and principal analyst of infrastructure and operations at Framingham, Massachusetts-based Forrester Research.
“When we speak to most enterprises, as well as hosting/collocation companies targeting them, the standard for average power consumption remains in the 4 to 6kW range. This is partly because few are engaging in wholesale IT swap outs. They have a mix of old and new, and appear to want consistency,” Villars says.
Others take a slightly more conservative approach. “For enterprise data centers, the average power per rack in the white space will remain in the 3 to 5kW range for the next 10 years at least,” says John R Laban, a London-based data center technology consultant.
Lest you take that solid 5kW as gospel, Laban may be simplifying the zeitgeist of data centers a wee bit. “Enterprise data centers (in-house or colo) are running at 3 to 5kW per rack on average. For racks containing only blades or converged infrastructure, you will see higher densities up to 10 to 12kW per rack, but that tends to be a small portion of the environment (for the average enterprise),” says Sophia Vargas, Forrester data center infrastructure analyst.
Assuming we wanted to build the perfectly optimized data center, what would we need to do? “With the high-density systems available today, it’s quite easy to exceed 15kW, or even 20kW, in a 42-rack unit (ru) rack,” says Steven G Hill, senior analyst in data center solutions at Current Analysis in Sterling, Virginia. Hill looked at the base numbers and pushed converged infrastructure projections even further.
“For example, a 6-ru blade system can consume 6kW alone, which averages to 1kW/ru. Even a rack filled with standard density 1-ru servers that average about 500w/ru would run over 20kW per rack, so imagining a data center that averages between 12-15kW/rack isn’t out of the realm of possibility,” says Hill.
Ultimately, imagining the realm of possibility and realizing the realities is a wide delta. It all comes down to the situation.
You must remember that working on electrical systems often means shutting off power and/or incurring major risk around power failover between phases. For most organisations this is untenable
“If you took the average of all existing data centers in the world, 5kW per rack is probably reasonable - there are a lot of older data centers under 5kW and a lot of newer ones that exceed that number by large margins. As far as new data center space, from what I have seen it is commonly built in the region of 8 to 10kW per rack, because one of the most common wiring schemes delivers approximately 9kW per rack. However, the density of the data center is heavily influenced by the intended use and loads, and by the fact that there is a ‘sweet spot’ in costs, with costs rising rapidly as density starts to go beyond 10kW per rack,” says Fichera.
Vargas cited diminishing returns of cooling efficiency as racks exceed 10kW, and Parks has real-world experience to draw from at Datapipe’s data centers. “We have found that going above 8.5kW tends to create a challenge in supplying enough cooling air to justify that type of density. In other words, we would simply lead the client to take another rack to spread out the heat load instead of increasing a rack’s density above 8.5kW. Above this number requires in-rack or in-row cooling techniques. Those tend to make the cost analysis not work favorably for the client versus just spreading the heat load out a bit.”
It’s not just the cooling problems that data center operators must contend with, there’s also the more science-fiction elements of converging that much raw power into a single geographical area.
“One of the key reasons why the mega cloud folks are resisting is that when one starts concentrating power in large complexes (as opposed to more limited HPC sites), there is an electrical effect that is the bane of the electric utility industry,” says IDC’s Villars. “Think of it as spontaneous, random static discharge, supersized. If you want a good visual, review the scene from [the perspective of the film] The Matrix, where Neo looks over the edge of his ‘pod’ and sees lightning arcing up and down all the towers. Unlike utilities, most DC operators don’t really want to ‘train’ for such phenomena.”
When it comes down to improvements, the sexy new data centers of the Amazons and Googles of the world just aren’t going to align with what we see in the data center down the street. Most CIOs simply don’t have the budget to make the kind of agile jumps that would increase rack capacity significantly. And while it works on paper, it’s unlikely we’ll see significant improvements in overall rack capacity greater than 5kW on average in the next five years.
Upgrade means shutdown
“You must remember that working on electrical systems often means shutting off power and/or incurring major risk around power failover between phases. For most organisations this is untenable,” says Greg Ferro, London-based network engineer and architect. “Data centers today have their existing electrical systems built around 5 to 7.5kW per rack, and unless a major refit or data center migration occurs there is insufficient reason to go through the pain of upgrading and modification.”
Data center operators agree that we no longer need to put on a sweater before venturing into the cold aisle. Industry standards are raising the thermostat on cold aisles, while still preserving fragile hardware.
“Many of our clients still believe the data center should feel like a meat locker when we take them on tours. That is old school and does not match the proven trends that dictate running the data center into the 80F+ [26C+] degree band that allows servers to run just fine and saves the data center operator large amounts of OpEx. And it’s naturally green to boot,” says Parks.
Whether optimization drives the average kW higher per rack, or whether the temperature experts can agree on any one thing, there’s never a dull moment in the data center arena.
But the interesting thing is that while market forecasters may be getting hot under the collar, the efforts of actual operators on the ground are keeping things in line. We’d bet that in future operating and management skills will keep the data center’s power needs under control