California-based healthcare company Kaiser Permanente is one of the largest nonprofit healthcare plans in the United States, with over 12 million members. The company operates 39 hospitals and more than 700 medical offices, with over 300,000 personnel, including more than 87,000 physicians and nurses.

Like many companies, in recent years it has been looking to reduce its infrastructure footprint. As part of a program to reduce IT waste, the company was looking to exit a leased data center and migrate any non-decommissioned systems to the cloud or a Kaiser-owned facility.

But when dealing with healthcare systems, access to data and applications can literally mean life or death in some cases. So ensuring that the company was able to exit the leased site without downtime was a key imperative for the company. For its efforts, the company won the DCD Data Center Operations Team of the Year award.

Kaiser Permanente first entered the Irvine, California, colocation facility in 2008 due to limited floor space and power capacity in its owned data centers. Eventually, the company decided to terminate its lease at the facility as part of its DCO Urban Renewal (UR) Program.

The UR Program was developed to find unused, under-utilized, abandoned, or partially decommissioned IT equipment that was consuming data center resources and decommission it. The company’s main goal was to close down the Irvine facility without any impact to Kaiser Permanente members or those who perform patient care.

The project encompassed migrating all compute environments to Kaiser Permanente-owned facilities and cloud providers, and/or decommissioning them. At the time the project started, the company’s footprint at the facility totaled 274 racks hosting just under 1,900 servers; the equipment covered 11,000 sq ft and required 1.4MW to power.

“Given the complexity of shutting down an entire data center, there were many roadblocks along the way. The risk of impact to the customers was extremely high,” according to Kaiser Permanente. “Every server connection had to be evaluated, every application studied, and every migration carefully planned.”

Before the team decommissioned the first equipment, there were several brain-storming sessions held with the core team members including compute, storage, network, application, design engineering, as well as non-technical members such as customer advocates, communications, and healthcare professionals.

The program sponsor was Elizabeth Burneko, executive director of data center operations at Kaiser, while Darren O'Toole, senior director of data solutions at the company, was the technical lead. The company also worked with IBM spin-off Kyndryl for the project.

During weekly planning sessions, the team developed multiple migration options for each application environment and jointly made the recommendations on which approach was best.

The teams then worked together to develop the step-by-step detailed migration plans; as each application was unique, the company wasn’t able to utilize a cookie-cutter implementation plan.

Each core team member had the right to object to anything in the implementation plan and voice their concerns openly.

“You can’t have a great plan without some healthy debates!,” the company said.

Given the healthcare-critical nature of the company, downtime was unacceptable, even during a decommissioning project.

“Since we provide urgent medical care to millions of members, there was no option to just shut a system down without studying every possible impact,” the company said.

In one scenario, after months of planning, the team determined that there was just too much risk involved in physically moving a storage system associated with approximately 80,000 users. Although much planning had gone into a “lift and shift” physical move over a weekend, it would have required an outage for all users.

Instead, after much consultation, the team determined that a no-impact, across the wire migration would be much easier for our user community, and were able to get all data migrated with no user impact.

One of the key challenges identified during the project was that end users were using their personal and shared drives to store information which could impact patient care if downtime was experienced during the migration. As a result, the company’s original plan to lift/shift some servers from Irvine to a Kaiser Permanente owned were changed on the fly to a network migration so no downtime would be experienced.

This took a tremendous amount of coordination and teamwork across multiple Kaiser Permanente organizations as well as our vendor partners,” the company said, but this network data migration approach has been utilized for other company efforts involving migrating large amounts of data across the network.

Kaiser said the project took “many long nights, weekends, and holidays to complete.” Just over 1,000 servers were decommissioned, resulting in more than $1 million through saved license and support costs, plus reduced need to build new data center space. The company saved more than $4 million through the termination of the Irvine lease.

As of August 2022, the Irvine data center was officially shut down with all compute and business operating equipment fully removed from the data center. And, crucially, the data center was shut down with zero impact to Kaiser Permanente members and employees.

On winning the award, sponsored by Excool, Kaiser’s Burneko said: “On behalf of the entire Kaiser Permanente organization we are very excited to be selected as the winner. It certainly did take a village across many arms of the Kaiser organization to accomplish the goal of closing down one of our colo facilities in Irvine, California. We are very honored to be selected by peers in our industry for this very prestigious award.”