Utility PG&E has cut power to hundreds of thousands of Californians in an effort to reduce the risk of wildfires.

Caught up in the rolling outages is the National Energy Research Scientific Computing Center, which has had to take its supercomputers offline due to the cuts.

NERSC is part of the Lawrence Berkeley National Lab, which is also mostly shut down - with the lab closed, JGI’s DNA sequencing suspended, the Molecular Foundry shut, and the Advanced Light Source inactive.

Update: NERSC and Cori are now back online.

NERSC goes offline

"PG&E has informed us that they will definitely be cutting power to the Berkeley Lab campus sometime between 12:01 am (Pacific) and noon Wednesday. Berkeley Lab is closed effective Wednesday, October 9 at 12:01 am. NERSC will continue to operate until power is cut by PG&E," user engagement group leader Rebecca Hartman-Baker said in an email to NERSC users ahead of the cut.

"When PG&E power is lost, NERSC will turn over to its generator. Cori draws too much power for the generator's capacity, so it will be immediately lost. Remaining auxiliary services, such as the NERSC website, will remain available until we can no longer maintain them on the generator (although we expect to receive fuel tank refills for the duration of the outage)."

Since the email, all of the high-performance computing facilities at NERSC have been shut down, including the 30 petaflops Cori supercomputer. A few core aspects of the center remain online at time of publication, after significant load shedding.

"NERSC is now operating on generator power alone til further notice," NERSC HPC systems architect Glenn K. Lockwood said in a tweet at 07:26 UTC 10th October. "Hats off to the teams up on the hill assessing the impact and watching rack temperatures tonight."

Update: At 17:45 UTC 11th October Lockwood tweeted: "Non-HPC resources at NERSC (incl. logins, gateways) are now going down too. All file systems being powered off, and anything depending on them (i.e., most user-facing services) will be offline."

He added: "Outage will be for days, but power's only been out ~11 hours. Now that daytime temperatures are rising and the generator-only infrastructure is stable, have to shed load."

Original article continues:

As for the return of HPC services, a NERSC status page states: "We do not have an ETA for services being restored. In a best-case scenario it could be restored over the weekend."

Update 2: In an email to NERSC users at 13:52 UTC 11th October, Hartman-Baker announced that PG&E has restored power to the substation providing electricity to the lab. "So today Berkeley Lab personnel begin the hard work of restoring the lab to normal, safe operations. This involves qualified Lab electricians and other experts systematically restoring power to nearly 100 individual buildings and then restoring power to individual pieces of equipment in stages, certifying that each piece of equipment is stable and safe to operate. This work begins at about 9:00 am (Pacific) today.

"The good news is that in the order of operations, restoring power to NERSC is the first item on the list! But once power is restored, there is much work to be done before we can return the systems to you. I will keep you informed of developments as they arise."

Original article continues:

NERSC's HPC systems are used by 7,000 scientists working on various research projects. "Our users run large scale climate models, they run large scale simulations of exploding stars, they run large scale simulations of a fusion model," Katie Antypas, division deputy and data department head at NERSC, told DCD earlier this year.

Just this week, NERSC detailed research it is working on simulating the 'Camp Fire' wildfire that last year killed 86 people and burned more than 150,000 acres. The research, recently published in the journal Hydrological Processes, used the HPC systems to understand how the fires impact California's watershed.

"Results from the high‐resolution model show counterintuitive feedbacks that occur following a wildfire and allow us to identify the regions most sensitive to wildfires conditions, as well as the hydrologic processes that are most affected," the abstract states.

Now, with PG&E trying to avoid a Camp Fire scenario reoccurring, such work has been paused, as have other research projects studying the impact of anthropogenic climate change.