At its worst, the February 2021 storm in Texas left 5.2 million people without power, while many others were about to find out just how high their uncapped, variable-rate power contracts could go. Data centers, chip fabs and other facilities dependent upon grid power also struggled to keep going - or were forced offline. On top of that, when the distribution system fails, a whole series of refineries, terminals and pipelines that criss-cross Texas and its neighbouring states go down at the same time. Economic damages have been estimated at $195 billion.

“This was the largest outage in the US since the 2003 North-East blackout,” according to Will Rohrig, senior vice president of Mission Critical Services at Foster Fuels. “And I was part of that one as well,” he admitted.

Rohrig was speaking alongside Dan Smith, a member of the Electrical Power Reliability Alliance (EPRA) advisory board, at the recent DCD>Grid Scale event.

“I started to see outages on Sunday afternoon,” continued Rohrig, but with only about 110,000 customers affected. “But we really saw it peak around Monday night, into Tuesday.” Even if the power only goes down for a short period of time, he added, in terms of the US fuel infrastructure it can have longer-term knock-on effects, both locally and across the US, “for weeks afterwards.”

He continued: “You can lose track very quickly how easily an ‘event’ in Texas can create a shortage of a particular blend of gasoline in Arizona.”

Expecting the unexpected

The especially inclement weather was not entirely unexpected, admits Smith. “People knew that the weather forecasts a week out were showing indications of pretty significant winter Arctic air. Then, as it got closer and closer, those forecasts looked more and more ominous. Admittedly, though, the duration and depth of the event were not really known until we were actually in it,” he said.

For electricity providers in Texas, Winter peaks look a lot like Summer peaks, with particular high points at around 6am in the morning and 7pm in the evening. Summer, continued Smith, peaks at around 75,000MW, and Winter peaks are typically around 10,000MW lower. However, because of the especially cold weather, coinciding with President’s Day on the 15th, the expectation was this year’s Winter peak would be similar to a typical Summer peak.

“At that point, we knew we were going to be short of capacity, so there were some expectations that load shedding [power rationing] would happen. A couple of days prior, ERCOT [the Electric Reliability Council of Texas] had been working with transmission operators, the companies responsible to execute a load shed. They have to work with the various service distribution providers to do that.”

Typically, top of the queue for 'load shedding' are major power users who are guaranteed lower prices in exchange for agreeing to be the first to be switched off, should the need arise. But when a significant megawattage needs to be shed the only option is rolling blackouts.

Initially, ERCOT was talking about shedding a modest 2,500MW. “Then they asked us for 5,000MW… then 7,500MW and many of us were like, ‘Well, that’s a pretty big ask. That’s not going to be easy to do’.”

So, added Smith, the projections were that the Texas grid was going to be about 7,500MW short.

On the night of Sunday 14th, coincidentally Smith’s birthday – “probably one of the worst birthdays I’d ever had” – it looked like the grid had made it through with minimal outages.

“I went to bed believing, okay, I'm going to get up early, and see how we're sitting for the morning peak – but we didn’t make it… Leading up to about 1.20am, some generation started to go offline, most likely due to the ‘winterization’ of some plants. But some of the generation [also] dropped offline.”

First, 1,418MW dropped offline, then 248MW, then 329MW over a period of mere minutes. When supply and demand are in equilibrium, grid frequency is stable at 60Hz. But with generation unexpectedly dropping, ERCOT ordered a load shed of 1,000MW to restabilise the grid.

Yet, almost straight after, 606MW, 688MW and then 511MW of generation rapidly dropped off, one after the other.

Today, added Smith, the reason for each and every outage still isn’t publicly known. “The initial outages were probably related to ‘Winterization’ [inadequate hardening to winter weather], but then from around 1.43am you start to see a bunch of generation trip offline, which was likely due to the decline in system frequency,” explained Smith. “It would be highly unlikely that all of those units dropped off at more or less the same time due to Winterization.”

In response to that sudden drop in generation, ERCOT ordered a further 3,000MW load shed just before the grid frequency hit a crisis low of 59.302Hz. The system was at below 59.4Hz – one of the system’s emergency thresholds – for four minutes and 23 seconds, according to ERCOT. If the Texas grid stays below 59.4Hz for nine minutes, generators would have automatically tripped out, making a bad situation for people shivering at home even worse.

At this point, according to media reports, the system was less than five minutes from a total black-out – “That’s probably true,” admitted Smith. He continued: “Not every generator is set to trip at nine minutes at 59.4Hz or lower, but many of them are, and more would’ve gone off.”

An additional load shed of 3,500MW, around four minutes in, helped stabilize the system sufficiently to enable the grid to return above the crucial 59.4Hz mark. This took total overnight load shedding to 10,500MW.

More than that, though, the system came within 0.002Hz of breaching the 59.3Hz mark – another trip-out benchmark. “59.3Hz is when you have automatic devices, called ‘under frequency load shedding relays’ and these automatically reduce load.”

In other words, if the grid had become any less stable, even more generation would automatically have been taken offline.

Overnight, the capacity of the Texas power system dropped from just under 70,000MW – during a period when users needed ever last megawatt – to 50,000MW. Output fell further throughout the 15th of February to just above 40,000MW, not starting to recover in earnest until the close of Wednesday 17th February.

Not going out

According to ERCOT, the biggest shortfall in generating capacity was due to outages at gas-fired power stations, accounting for just over 10,000MW of lost output, most of which was unscheduled, according to Smith. Wind generation was the next biggest energy source posting outages. “There wasn’t a resource type that didn’t have an issue apart from hydro, and hydro is a very small percentage of capacity in the ERCOT market, although it’s an essential one for certain ancillary services,” said Smith.

Even coal-fired power stations suffered some modest, unplanned outages, while the South Texas Nuclear Plant also chipped in, although that was the only outage contributed by the nuclear power sector in Texas.

For data center operators and other organizations (such as hospitals) with diesel backup generation, one of the additional challenges they faced was not just getting hold of fuel should outages last more than a few days, but the fact that, untreated, diesel starts to freeze at between zero degrees celsius and about -9.5 degrees – and diesel fuel in Texas is typically untreated.

On top of that, when diesel is on the cusp of freezing (and that point can vary by a few degrees either way, depending on the batch) it’s not always easy to see when it’s turning to jelly, warned Rohrig, particularly with ultra-low-sulfur diesel, which is more vulnerable to cold weather.

“The term that most people are familiar with is ‘gelling’. Gelling is a process where the paraffin wax crystallises in the fuel. Gelling starts to occur in about the 20 or 25 degree [Fahrenheit] range. Each batch of fuel has its own specific point where it can start. This gelling process is clearly visible in the form of a waxy deposit on diesel generators’ fuel filters.

However, when diesel is subject to ‘icing’ it’s not nearly so clear.

“Icing can start as early as 30 degrees [Fahrenheit]. Icing is just what it sounds like: it's the water entrained in the fuel that starts to freeze. But on the filter, it doesn't look like anything, it just looks like a clean filter. So it's hard to spot; it manifests itself as filters that plug-up for no other apparent reason,” said Rohrig.

The lack of cold-weather treatment for diesel in Texas reflected a wider lack of ‘hardening’ against cold weather across much of the state.

“Facilities, especially in South Texas, are not hardened against 20-degree temperature swings. What might be normal from a winterization standpoint in the North East [US] would be completely unheard of in Texas,” said Rohrig. “[In particular] we were not recommending winter operability treatment for diesel fuel in Houston. It just didn’t make financial sense. Obviously, in the wake of this, we’re re-evaluating that recommendation pretty heavily.”

As a result, Foster Fuels had to resort to moving fuel from 12 different states into Texas in order to keep customers going.

Also a significant challenge, according to Rohrig, was simply employee availability at facilities such as oil terminals and data centers. “They were just as impacted as the rest of the facility itself. Certainly you need that you need people to be there to run a facility and they're dealing with family issues and then road hazards and that kind of thing.”

Concluding the presentation, Rohrig and Smith covered recommendations for the future, from something as simple as straightforward communications with the public and customers, to ‘hardening’ the Texas grid and improving load management – and after that, the industry can look forward to some legislative and regulatory ‘hardening’, too.

Missed 'An Anatomy of Grid Failure: What Happens When Reality Bites'? Watch the full presentation for free, on-demand NOW!