This feature appeared in the April issue of DCD Magazine. Subscribe for free today.

Colocation buildings started out as neutral territory where enterprises and other organizations could rent space, bandwidth or equipment. Most customers would need a rack or two, or maybe an aisle. They’d get a pass to get into the facility, and a key to the cage and their rack(s).

Sometimes called “carrier hotels,” these facilities were an opportunity for organizations to close their own data centers, or to expand beyond IT suites that were bursting at the seams, using space, security, power and cooling that was paid for by a central provider.

This “retail colocation” approach is still very significant, but in many locations, it is being superseded by wholesale colocation in which a giant customer, usually a large enterprise or a web services provider, buys up a whole suite, a hall or even a whole data center in one go.

“Wholesale is massively outgrowing retail,” said Adil Atlassy, recently appointed as CTO at wholesale provider Compass Datacenters. Previously, he managed site selection for Microsoft, handling the giant’s acquisition of data center space, so he’s seen the change from both sides of the transaction.

Wholesale colo providers vs hyperscalers

In volume terms, retail may be growing, but the customers are changing, Atlassy told DCD. Customers are migrating much of their IT to the cloud, but their “crown jewels” remain in their own data centers, he said. This creates a hybrid cloud. “A tremendous amount needs to be optimized in the cloud to move the crown jewels,” he points out. But the massive growth in the public cloud equates to a boom in the wholesale space the hyperscale cloud providers buy.

Compass builds in multiple sizes, with 1.2MW and 2.4MW buildings suiting retail colo providers, and enterprises looking at 6MW facilities. For hyperscalers, Compass makes 32MW data centers.

In Northern Virginia last year, most of the colocation providers DCD visited were entirely focused on chasing the wholesale market, with very positive results. Big customers, up to the size of Google or Facebook, were buying up capacity faster than they could build it.

“We believe our business works best when we have an intentional focus on the hyperscale community,” Tag Greason, chief hyperscale officer at wholesale colo provider QTS, told us at the company’s new facility in Ashburn. It’s obvious where that’s taken QTS: the current facility is three stories tall, and will have 32MW of available power when complete. It is mostly being leased a room at a time, but those rooms are big.

The size is perhaps more obvious at Digital Realty. At 36MW, Building L in Ashburn will be Digital's biggest - and that's just the first phase of a facility which will ultimately offer 84MW. Building L will have 6MW data halls, each of which is 36,000 sq ft, just short of an acre (43,600 sq ft). At that size, the halls have diagonal pillars to brace the ceiling against earthquakes - even though Ashburn has no seismic activity.

“Only five years ago you felt a 250,000 sq ft building with 10MW was a lot,” said Jon Litvany, senior sales director at Digital. “Now we may do a lease for 36MW for one client, and they'll take it in six rooms.”

Most players keep a foot in the retail colocation space: “Few people out there are pure play. Everybody's got a mix of public cloud and infrastructure and bare metal, and that mixture falls into a bucket called hybrid colocation,” said Greason.

So wholesale colo providers often have a hall or two of retail colocation which, these days, is sold under the hybrid brand because it is capacity which has come out of an in-house data center, and may be moved on to a cloud provider, but is run by a customer using all three.

Wholesale colo providers have one big competitor when selling to hyperscale customers: the hyperscalers themselves. Hyperscalers build their own giant, cheap facilities for planned capacity. They shop around for wholesale colocation when they have to top this up.

“They build this 200MW campus, and then they look to me for one of three things,” said Greason. “Bringing capacity on a little quicker, providing overflow for production capacity that they didn't plan for, and moving non-production workloads out of their environments.”

Atlassy thinks wholesale colo can get more of the hyperscale business: “The volume at which [the cloud giants] are all growing dwarfs their ability to provide in-house. To make that growth, they will have to outsource,” he said. His belief is that ultimately, Facebook, Google and Amazon Web Services (AWS) will decide that construction is not their core business.

Those chasing hyperscalers need to come close to matching the costs the hyperscaler would pay to build for themselves: they call this “owner economics.” Said Greason: “They're coming to me because I'm close to owner economics.”

Atlassy agreed: ”You want faster, cheaper and better,” he said, and traditionally, the customer has to choose any two. He reckons the industry can get beyond that - and for him, the secret is scale: “With scale and volume comes more efficiency.”

Innovation for hyperscalers is about getting efficiency while taking out as much cost as possible, without jeopardizing uptime. Providers aim for between $5 million and $10 million per megawatt - that's a wide variation, but then there is no clear standard for what is included in that price, and how long the contract is for.

It starts with a focus on the building: “Hyperscalers care about location, economics, scale and speed. If you can't answer the mail on those four, you are literally not at the adult table talking with them.” So if Facebook asks for 10MW in Ashburn, it doesn’t matter how good your Atlanta data center is: you don’t get the deal.

To do this, they also have to be physically close to the giants’ campuses.

Church and state

Moving non-production workloads to wholesale colo leaves the giants with a homogeneous IT load of customer-facing applications in their own monolithic facilities, said Greason: “Hyperscalers can be very, very efficient in their operations, if they don't have to worry about test-dev, IT, or non-production assets within their production environment.”

And those non-production jobs are very significant. Greason reckons around ten percent of a hyperscaler’s workloads are non-production. For a smaller company, that wouldn’t amount to much, but for a Google or an AWS, it could be 10MW.

“They're dealing in hundreds of megawatts in capacity planning - and 6MW is a rounding error,” said Greason. “'Can you take care of this little nuance?' For me and my competitors, that's not a rounding error. A regional financial bank wouldn’t say 6MW is a rounding error. It's their entire business. It's probably double their entire business.”

For wholesale colos, dealing with the hyperscalers’ overflow isn’t lesser work for Greason: ”Five to 10MW of non-production environment is very valuable to me - and it’s very valuable to them to move it.“

Wholesale colo must also have scale: “You must build big, but can you manage it?” asked Greason. “And do you have experience doing it?” The big wholesale colo players all point to their track record.

The final requirement is speed, but this is not a simple measure, said Greason: “Most people just immediately think, how quickly can you deliver? But how fast can you do the lease? How fast can you answer the RFP? How quickly could you purchase that land in a new location? There are so many aspects: it's not just about bringing a megawatt online.”

The last part is speed of construction. Wholesale colo providers build the shell first, and then fill out the floors.

The order of construction is important. QTS is building 32MW on three stories, so it fits out one 8MW quadrant at a time. Part of the reason for this is that the cooling and power systems are shared across all three floors. One quadrant’s worth of mechanical and electrical infrastructure is installed, and then the three floors are filled up.

Digital Realty builds two-story facilities and fills the top floor first. Moving heavy equipment in on the ground floor is less likely to have an impact on an upstairs floor than doing it the other way.

All modern colocation vendors put mechanical and electrical equipment outside the IT halls. This is a major plus, as it allows a separation between the shared infrastructure managed by the provider and the equipment installed by the customer. It’s a separation of “church and state,” Jon Litvany told us, standing at the door to an IT hall in a Digital facility: “We own everything out here, the client owns everything in there.”

That division also allows the provider to keep its own infrastructure at a different temperature to the customers’ - important when “wet-cell” VRLA batteries are in use.

One common piece of the answer is standardization and commoditization. Big colo providers have standardized their buildings, making the same shells in different locations. They also have pre-configured quantities of mechanical and electrical hardware they buy from vendors In QTS’ case, the modular design is called QMOD: “We standardize on the UPS and the generators, and we have the ability to just buy equipment and move it around to different sites.”

“We look at components that are really widely available and commoditized,” agreed Atlassy. But they are carefully selected to match, so there is no stranded capacity.

Theoretically, these standard purchases could be with multiple vendors, but in practice, each colo vendor has preferred suppliers: “We design it once, we have one set of suppliers, and we build that relationship with them to make sure that we have the right lead times, and the right delivery windows.”

As well as standard components, providers use pre-fabricated construction where possible, in particular assembling mechanical and electrical components as “pods” and creating the building with “tilt-wall” construction (see below).

The benefits of pre-fab are obvious: “you shrink your timeline, and you increase your reliability,” said Litvany. Buildings can be completely built, and filled with customers in less than one year.

Two concepts

But providers admit an apparent contradiction with this model. Doing business is about giving the customers what they want; so how do you square that with a limited offering? It turns out, that is what the customer actually does want, they all told DCD.

“Five years ago, when everyone envisioned going to hyperscale, they said, ‘Let's just go with a blank slate, let's give them whatever they want!’” said Greason. “Over time, the providers found that hyperscale customers came to them with specific standard designs, and then started to consult with the providers on the details.

“I think the hyperscale conversation has ended in two concepts,” said Greason. Originally, some hyperscalers would specify the design and the equipment; more often these days, he said “Vendors say, ‘We don't care what you use - within reason.’ They don't want to disrupt our timing and our schedule, because all they want is a quality product delivered on time, at scale, and with near owner economics. If they start monkeying with all of our components, the cost goes up, and the delivery time gets later.”

Some choices are easier to introduce at a large scale. For instance, many hyperscale customers are going with slab floors for their own facilities. Aisle containment systems can be built on top of a solid concrete base, and the cost of strengthened raised floors is eliminated.

When you sell a building at a time, it’s easier to offer both. The next building in the long line at the Digital campus will have slab floor at the customer’s request.

You get an idea how much the customers like limited choice, when you hear how hard the operators have to work to introduce changes. For instance, when QTS built to three stories, the hyperscalers had to know that they could deliver their equipment in heavy racks to the site and get it all the way to the IT floor - that lifts and doors were all sized for the hardware.

In wholesale colocation, an inventive set of suppliers is dealing with a demanding and exacting group of customers.

It’s a small world, in terms of the number of companies involved, but those companies are finding new ways to put together a giant and rapidly expanding network of massively powerful infrastructure.