Microsoft is testing out an immersion-cooled server deployment at a data center in Quincy, Washington.
The two-phase cooling system was developed in collaboration with Taiwanese server manufacturer Wiwynn.
For breaking data center news, features, and opinions, subscribe to DCD's newsletter
A different kind of submerged server
The system uses tanks in which electronics are cooled with engineered fluid from 3M, which has a boiling point of 122 degrees Fahrenheit, 90 degrees lower than that of water.
The fluid turns to vapor, rising to meet a cooled condenser in the tank lid, where it turns back to a liquid and falls back onto the immersed servers. The cooling coils are connected to a separate closed loop system that uses fluid to transfer heat from the tank to a dry cooler outside the tank’s container.
“We are the first cloud provider that is running two-phase immersion cooling in a production environment,” said Husam Alissa, a principal hardware engineer on Microsoft’s team for data center advanced development.
Microsoft appears to be the first western cloud provider, but Alibaba began testing immersion cooling systems in 2017.
In a 2019 OCP video, senior engineer Yangfan Zhong said that the company had run a large-scale deployment for the past year, and expected to build a second immersion cooling data center with 20,000 servers by 2021. Outside of cloud providers, there are also numerous companies researching or offering single-phase or two-phase immersion cooling, among them GRC, Iceotope, Asperitas, and Submer; some using 3M fluid, others cheaper stuff, and some proprietary liquids.
The idea dates back to the earliest days of computing, with some of the first supercomputers submerged in different fluids, but fell out of favor when air cooling proved sufficient for systems at the time.
But companies have turned back to immersion cooling as chip densities and power use has grown. “Air cooling is not enough,” said Christian Belady, distinguished engineer and VP of Microsoft’s data center advanced development group. “That’s what’s driving us to immersion cooling, where we can directly boil off the surfaces of the chip.”
Together with undisclosed industry partners, Microsoft will study how to run the tanks in ways that mitigate fluid loss.
In a blog post announcing the project, Microsoft notes that "several technology companies, including Microsoft, are [also] experimenting with cold plate technology." In fact, the supercomputing industry has seen numerous companies offer cold plate technology for years, while Google's TPUs appear to rely on them for cooling, but it is not used in standard hyperscale deployments.
With its immersion system, Microsoft hopes it will see the same hardware reliability improvements seen in its Project Natick test, which put a data center under the sea. Should that prove so, the company expects not to immediately replace components when they fail, limiting vapor loss as well as allow tank deployment in remote, hard-to-service locations.
The system could also prove useful in Edge deployments, allowing a large amount of compute with a small fooprint.
But before then, Microsoft plans to test the single tank for the next few months.
“This first step is about making people feel comfortable with the concept and showing we can run production workloads,” Belady said.
