I am guilty. Let’s get that right out of the way. I am guilty of allowing too many of my clients not to implement waste heat recovery in data center cooling system designs. The reasons throughout the years have been somewhat understandable; the business case has not been viable enough, there have been security issues, worries about mission critical cooling stability, inconvenient refrigerants, lack of investment funding, reluctant district heating utilities, lack of government strategic support…

Those reasons are, as we say here in Finland “Snow of the winters past”. We Finns do like our snow very much and it’s starting to show in what we do, with our government leading the way.

The Finnish government has not only offered the usual stick in form of increased taxes on fossil power plant fuels but also a big carrot for data centers. The carrot, scheduled for implementation in 2021, is a huge electrical tax discount for data centers that supply their waste heat to district heating utility networks. Up until now the lower tax class has been for data centers exceeding 5MW.

Case study: Old paper mill becomes a green data center

Kajaani is an archetype of a Finnish industrial city, located about 450 kilometers north of Helsinki, inhabited by 40 000 people and surrounded by vast forests. It was forest industry that gave the city its wealth and source of livelihood since the early 20th century.

When digitalization put an end to the golden era of paper manufacturing, the forest industry giant UPM had to shut down two remaining paper machines in Kajaani just before Christmas 2008. The city and people of Kajaani and the owners of the huge former paper mill site “Renforsin Ranta”, the “Renfors Shore” faced a new challenge which they took on quite decisively.

CSC, the Finnish IT Center for Science a is a non-profit state enterprise that is mostly focused on high-performance computing (HPC) services for science and education. CSC entered Kajaani and Renforsin Ranta in the early 2010’s and commissioned its now-retired ”Sisu” supercomputer in 2012. By that time, it had become evident that the former UPM Kajaani paper mill site was a near-ideal brownfield location for HPC applications, thanks to its cool climate, robust electrical infrastructure and well-maintained, sturdy industrial building shells. Renforsin Ranta has attracted also other data center operators over the years.

LUMI (the abbreviation stands for “snow” in Finnish) exa-scale supercomputer consortium, formed by eleven European countries and led by CSC, entered The European High-Performance Computing (EHPC) Joint Undertaking officially in November 2019. At that point, the project team had been silently working in the background for more than a year.

From the very beginning, premium energy efficiency and 100 percent waste heat recovery and recirculation were identified as something that set Kajaani apart from the rest of the EHPC joint data centers, both from the perspective of operating cost and climate change mitigation.

We like our houses warm and climate cold

The Nordics are big in district heating. In Finland district heating covers almost half of the needed building heating capacity. Sadly, though, still a third of the energy comes from fossil fuels.

The lowest winter temperatures in northern Finland can easily be lower than -40C, and I can assure you this feels cold. The majority of district heating production for the city of Kajaani takes place in a classified peat and wood-fired cogeneration power plant located in Renforsin Ranta. So the district heating infrastructure was not far from the old paper mill hall that was selected as the supercomputer location, a fact which made the choice of a recovered heat consumer very obvious.

Loiste, the local district heating utility greeted the data center heat recovery with enthusiasm and a can-do approach from the very beginning. Loiste is a perfect example of how Finnish district heating utilities are opening their networks for new opportunities. Replacing as much slowly renewable peat fuel as possible with waste heat was a perfect fit for their strategy.

In a way, the battle against climate change in the field of refrigeration technology has opened new opportunities for waste heat recovery. Certain 4th generation hydrofluoro-olefins of “HFO” refrigerants, developed for negligible ODP and GWP turned out great for heat pump applications, not to mention that they are safe to use as well. Technical development in the field of HFO refrigerant and water-cooled chiller technology -based heat pumps has significantly lowered the initial investment costs of an industrial-scale heat pump system that can produce hot enough water for most applications, district heating supply water in this case.

In wintertime, the entire water-to-chip-cooled LUMI supercomputer is cooled by three HFO screw compressor heat pumps which provide a total maximum of 9.5MW of heat to the Loiste district heating network and buildings in Renforsin Ranta area. The LUMI HPC site uses carbon-neutral electricity produced by renewable energy sources and the annual 60GWh of district heating energy it produces, has reduced the CO2 emissions of Kajaani's city-wide district heating by almost 12 000 metric tons annually. As a reference, Loiste sells 280GWh of district heating energy annually and has 1700 customers.

We took a road less travelled

How does a district heating utility network serve as a cooling heat sink for a mission critical data center, then? Although Finnish district heating networks are usually very reliable, the primary heat rejection from LUMI HPC is designed by dry air coolers to ambient air – a solution which yields an impeccable free cooling PUE (power usage effectiveness) of 1.05.

To squeeze every bit of efficiency from the heat pumps, they’re connected as water chillers to cool the racks directly. Should the heat pump system fail, the cooling load swing to ambient air rejection is a rather drastic measure which needs to be designed and tested carefully to guarantee stable cooling supply. Air-cooled server data centers are relatively straightforward to heat-load test in the commissioning phase. In liquid-cooled applications, though, the test arrangements are a bit more complicated. To limit the extent of actual, cost-heavy on-site commissioning tests, we took a road less travelled.

The whole HPC cooling system was modelled as a Digital Twin in the Apros process simulation software, which is widely used in nuclear industry and other critical applications for design validation and transient analysis. Transient simulations, carried out by our partner Semantum, have already provided useful information for the design phase in terms of thermal storage size validation and operation mode automation sequence timing.

However, the possibilities of a Digital Twin, once established, are not limited to just transient and failure consequence analysis or thermohydraulics. An entire array of commissioning tests can be run virtually before the actual tests. All extensions and changes to the system may be simulated before putting them into action. A Twin can also be used for energy optimization and balancing.

The design phase of the LUMI EHPC project is now complete, and construction works on site are ongoing. When commissioned, CSC’s LUMI will be in world’s supercomputer top 10 by with its tremendous 200 Petaflops of calculation capacity.

The cool number-cruncher needs an equally cool infrastructure around it to keep LUMI running in the most high-quality, cost-efficient and environmentally sustainable way imaginable. Heat recovery business case calculations demonstrated that data center operators, at least in The Nordics have no excuses for not putting the waste heat to work these days.