"We really weren't afraid going into it. In fact, we were live streaming it. It was exciting… up until the point where the surge hit."

Hurricane Sandy was merciless. In 2012, the terrifying storm killed hundreds and cost billions across eight countries. In New York's financial district, home to a cluster of data centers, it proved equally devastating.

"I would do data center tours all the time, and I would show off about our redundant fuel tanks, redundant pumps, wired to separate power sources, and the generators and all that. It sounded really good," Alex Reppen recalls.

"But when the storm hit our basement it dislodged our fuel tanks, it was so powerful that the rush of water coming in through the grates on the street was enough to rip the bolts out of the concrete."

At the time the founder and CEO of Internet service provider Datagram, based at 33 Whitehall, Reppen got to watch his flagship data center falter before his eyes. It took down big-name sites including Gawker, Gizmodo, Buzzfeed, and Mediate.

This article appeared in Issue 40 of the DCD>Magazine. Subscribe for free today

The surge

Due to regulations enacted after 9/11, New York data centers keep their diesel fuel in the basement, and their generators on the roof. This is fine in normal outages, but proved disastrous in Sandy.

With the Datagram site offline, "on the evening of the surge, we were in the streets with our electricians building makeshift pump controllers on plywood and holding them up 12 feet up in the air to keep them dry," Reppen said.

"At the same time, our building was working on pumping out the basement which took a very long time. We really didn't even know what we were dealing with - a couple of guys went down in the water, and when we found that it was unsafe we were forbidden from going back down there until it was pumped out."

Once it was finally pumped out, they discovered that the building's electrical riser was destroyed from the bottom. "So we had to run cable up 25 stories just to get to a street generator," Reppen said. "Once the street generator arrived, we put it on the new main riser that we built. And then that generator failed because it had dirty fuel."

Fuel vendors, seeing opportunity in crisis, sold whatever they had on hand. Heating fuel was sold as generator fuel. "We got new generators, we had about three generators at one time on the street. I was down there with buckets of fuel trying to filter them out and it was a nightmare," Reppen said.

For Datagram, the disaster did not alleviate as the storm ebbed. The sheer scale of the damage to the building and - Reppen claims - a slow and difficult landlord, meant that the company ran its data center off of diesel for three months. "They just took their time fixing that electrical riser, which you don't realize how dependent you're on until it gets flooded."

Running off of a generator for so long was a deeply exhausting and fraught experience, he recalled. "I lived like six blocks away, and I remember even a month later getting the alarm at three in the morning, and then I'd be sprinting as fast as I could.

"You'd be looking to see if every light was off. It was just such a nightmare," he reiterated.

Many of the staff slept at the data center helping keep it going, or communicating with customers. "We didn't have customers immediately canceling," he said. "We had little $29 and $100 a month customers screaming bloody murder and canceling, but the customers who paid $1,000s and tens of $1,000s a month were fine. They understood that this is a disaster."

Adding to the pressure, Datagram was in the midst of potential sale discussions, ultimately being acquired by SingleHop in 2015 (the site is now owned by INAP). "We probably lost tens of millions in potential valuation," Reppen said.

"We also lost a ton of money just paying for diesel fuel, paying for customer credits, paying for moving customers. Building up a 25 story copper riser alone was $400,000-500,000, just for the copper, which then the building got to keep."

The company ultimately sued the building in "a five-year lawsuit that really went nowhere," he said. "We sued our insurance company and went into arbitration. [Mayor] Bloomberg reached out to us and offered us help and assistance, but it turned out to be nothing. It really was not great."

The legendary bucket brigade

Datagram was not alone in its struggle. The chaos took out other providers in Manhattan, Internap, Steadfast Hosting, Init7, and Cogent, among others. A few managed to survive, notably one which turned to a now-legendary solution - the bucket brigade.

Less than 200 yards away from Datagram, Peer 1 had a similar issue in a carrier hotel at 75 Broad Street. Most of its fuel storage was underwater, except for a small diesel tank by its generator. Thanks to Peer 1’s small size (400kW), this equated to about six hours of data center runtime.

So, the company figured, as long as it could get the diesel fuel up to the generator it would be able to keep running. At first, the team tried to carry 55-gallon diesel drums on hand trucks one flight at a time - for 18 flights. "That wasn't sustainable," Michael Mazzei, the data center's manager at the time, told DCD.

"So you go through Plan A, Plan B, Plan C, and then when you start getting further into the alphabet you actually start coming up with some plans that work," he said.

"What ultimately worked for us was transferring the diesel into smaller five-gallon barrels and carrying those up."

The team, building staff, and even customers like Squarespace and Fog Creek all chipped in. The bucket brigade carried on for sixty hours.

Mazzei used other techniques to extend the useful life of the fuel tank: turning off the CRAC units and air conditioning in the Peer 1 space, and allowing the temperature to rise.

"I'm terribly out of shape, and after doing just one of those trips I was really wondering if I was going to be one of those young people that was going to have a stroke," staff member Jeff Burns said in a promotional Peer 1 video a year after the storm.

Peer 1’s one-time neighbor Reppen is critical of the effort, telling DCD "at the time we heard the stories of the guys next door carrying the buckets up and slipping downstairs and everything.

"I mean, I'll be honest with you, I walked up the stairs once or twice. I was dead, and I'm a fairly in shape guy. With the humidity and everything going on, I couldn't imagine going up [18] flights with buckets of diesel."

He added: "It wasn't worth the risk, and that was stupid. I'm surprised somebody didn't die."

Mazzei counters that the team was careful and "smart about it. Diesel is a safe fuel from the perspective of transportation," he said. "I don't think we would have done this with gasoline, just because of a light bulb dropping... diesel is a very stable fuel, you have to atomize it, and you have to compress it for it to be problematic.

"It was just one of these things that, you know, you get initial momentum behind it, and it just kind of acted like a flywheel and we kept going. You're running on adrenaline, you know that there is a working running data center and it was vital to Squarespace as well as other customers."

Keeping the facility live was also deeply personal for Mazzei. "I felt like as much as it was a Peer 1 data center, it was a data center that I actually started from day one. I just felt that if I could keep that data center running for my customers, I was ready to go the extra mile. And we obviously had to make it through a lot of obstacles. And we did."

Plus, he argued, the facility was able to "provide a lot of good.” With no working phones in the building, and mobiles not connecting, people would come to the data center to make calls. “We had all sorts of vendors that were helping the building coming up to the second floor and using our open WiFi so that they could communicate and get emails out.”

The bucket brigade worked, and Peer 1 stayed up throughout, while other providers in the same building, including Internap, went dark. "Fortunately for us, no one got hurt, we had no mishaps," Mazzei said.

"We were spilling a little bit of diesel fuel on a sidewalk here and there, of course," he admitted. "But it made me laugh - we had people complaining [about the spills], and at the same time we had cars floating past, gasoline and diesel from the floods, and sewage. You're complaining about five gallons of diesel that I might have spread here and there?"

Nine years on, from the comfortable vantage point of hindsight, it's hard to judge the actions taken at the height of the storm.

Given a similar situation happening at the time, "I would have done it again," Mazzei said, but added that with most customers now spread across multiple data centers he wouldn't do it in the current climate.

Still, even with all that effort, the whole endeavor could have easily collapsed.

"All it would have taken was one clogged fuel filter, and it would have all been for nothing," Mazzei said. "We ran a single generator for 10 days, and it felt like forever. Generators are amazing pieces of equipment but you're just one fuel injector, one piston, one belt, one hose away from bye-bye"