A 'fluid' way to cool high-density data centres

  • New immersion cooling technology drastically cuts energy and space costs
  • Key to technology are new techniques and new super fluid by 3M 
A 'fluid' way to cool high-density data centres

IT was an experiment born out of necessity, when solutions provider Allied Control was tasked with building a high-density data centre in Hong Kong, known for its extremely high real estate costs.
Due to the hot and humid summer climate, the team also had to deal with average annual industry power usage effectiveness (or PUE, a measure of how efficiently a data centre uses energy) of 2.2, according to a study by Hong Kong’s biggest electricity provider, China Light and Power.
A 'fluid' way to cool high-density data centres In order to decrease operational expenses, the team considered the latest available data centre cooling methods for high performance computing (HPC) applications, but were still not satisfied with the results.
“Furthermore, studies claim that around 2,000 cubic feet of air are required to adequately cool a single high-density server rack of 20kW," said Kar-Wing Lau (pic), vice president of operations for Allied Control Limited.

"That’s equivalent to a staggering 56km/h of wind speed when forced through a one-foot wide air duct, which is similar to placing your hand out of the car while driving.

"It was not feasible for us to waste that much energy and costs just to provide the necessary cooling. We also found that no single vendor was offering what we needed,” he told Digital News Asia (DNA) via email.

The team found itself with “no other choice” but to experiment with 'evaporative two-phase immersion cooling technology,' by fully immersing electronics into dielectric and environmentally sustainable 3M Novec fluids.
“No one has ever been known to use this technology in a large-scale data centre implementation before,” Lau said.
3M Novec Engineered Fluids are a family of proprietary non-flammable fluids that are being applied for a number of industrial use cases, including precision cleaning and heat transfer.
“It was really an immense challenge, but we set up our 500kW immersion-cooled data centre in only six months from conceptual design to completion. This included securing even the data centre site and the entire supply chain.
“To make this possible, we had to temporarily upsize the project team to a total of 25 employees. A typical data centre of this size often takes at least double the time to be set up,” Lau claimed.
Using the two-phase immersion cooling technique, the team claimed savings of 99% on cooling electricity and 87% of space compared with traditional air cooling options.
The team’s efforts won them recognition in the form of the “Future Thinking and Design Concepts” award during Singapore Datacenter Week, which was held Oct 13 – 17.
How it works
A 'fluid' way to cool high-density data centres Lau said that efficient cooling is about navigating intelligently within the physical limits of heat transfer. Liquid cooling, like water-cooling, is much more efficient than air-cooling and allows for higher densities at lower costs.
“But since water and electricity don’t mix well, a physical separation via custom-made metal cold plates containing an intricate system of small copper and plastic tubes needs to be implemented, which quickly becomes out-dated with every additional chip and socket generation due to changing form factors,” he said.
Additionally, with rather messy oil-cooling methods that make the hardware greasy, the heat transfer coefficients for forced liquids through pumps in small areas around chips are also limited, he added.
With two-phase immersion cooling, the heat of the completely immersed electronic components causes the fluid to boil at relatively low temperatures of 34° to 56° Celsius, depending on which fluid is used.
Heat is then transported away with vapour contained in the bubbles that rise through the fluid surface, up to the second cooling loop with a bigger condensing coil containing normal facility water below the fluid’s boiling temperature, where it condenses while transferring heat to the facility water.
The condensed fluid then falls back into the tank containing the electronics at a cooler temperature. Due to the turbulent boiling, no pumps are necessary in the first cooling loop and it provides cooling even to the smallest areas.
According to Lau, the advantages of this technology are multi-fold. Firstly the facility water doesn’t have to be as cold as that of chilled water supplying the Computer Room Air Conditioner (CRAC), which is usually at 7° Celsius.

This allows very low energy expenditure to cool down facility water, even at hot and humid ambient climates.
Due to the efficiency in heat transfer even at the smallest chip level, and in combination with centralised, larger condensing coils instead of many small copper tubes of normal pure water cooling systems, the system only requires a relatively low water flow, which can save a lot of pump energy.
“This, in combination with the first point, leads to cooling electricity savings of 99% and a PUE of 1.01,” said Lau.
Lastly, as all electronic components are being cooled uniformly and from all sides, no heat sinks or water cold plates are required.
“By getting rid of all bulky heat sinks and being able to stack mainboards with only a few millimetres in distance from each other, this allows us to achieve unprecedented densities and thereby save a lot of space,” he added.

Next page: Addressing concerns and what's next 

A 'fluid' way to cool high-density data centres

Addressing the concerns
Lau was quick to point out that the two-phase immersion cooling system is intended for high-density HPC applications. If a data centre is only working with low-powered webservers, then the recommendation is to stick with air-cooling.
“With high density HPC applications, the amount of energy and costs required to use air cooling becomes extremely high. Also, not every data centre operator can choose to set up in a colder climate like Iceland to reduce cooling costs, especially in Asia where warmer and more humid climates dominate.
“So it comes to no surprise why PUEs are generally higher [here] than, for example, in North-America or Europe,” he said.
Use case aside, many commentators have expressed hesitation over the need to change the layout of equipment within a data centre, as current designs are based on vertical racks rather than horizontal ones, as required by the liquid immersion cooling technique.
There is also the concern over the added messiness involved with swapping out hardware, given the presence of liquid.
Lau welcomed the opportunity to address these “common misconceptions," noting that the company’s second generation design which won the DatacenterDynamics APAC (Asia Pacific) Award and Best Green ICT Award, was actually for a standard vertical rack deployment up to 225kW per rack.
“Technically, there is no fixed requirement to use horizontal racks instead of vertical ones, or to change equipment layouts. However, our next generation system will use flat rack tanks for improved efficiency by scaling to 500kW per rack, which would be difficult with vertical racks. Not everyone requires such densities and therefore horizontal racks,” he argued.
A 'fluid' way to cool high-density data centres As for the perceived messiness involved, Lau agreed that this would be the case for oil immersion cooling, but Allied Control's system uses the 3M Novec fluid, and hardware comes out touch-dry almost instantly when pulling it out.
“At the Supercomputer Conference SC13 we immersed a standard US one-dollar paper bill for hours in 3M Novec and demonstrated that the one-dollar bill became immediately dry and was still like new when removing it from the fluid,” he claimed.
The fluid is also a high-precision cleaning agent and a fire-extinguishing agent, with Lau noting that it is so sensitive to delicate materials like paper, it has been installed in the US National Archive, home to the US Constitution and Declaration of Independence.
“Since the very same fluids are used to clean the inside of mobile phone screens or even hard disk platters, where even microscopic residues could cause head crashes, there is absolutely no reason to be afraid of messiness of any kind.
“We have even sold out-dated hardware which had been immersed in the liquid for about 10 months and customers were genuinely surprised, claiming they thought we were selling brand-new hardware to them,” he added.
Lau also pointed out that there might be much more accumulated dust and other dirt on hardware coming out of traditionally air-cooled systems, possibly even corrosion if air is not filtered properly in areas with high air pollution like in China and India.

In terms of maintenance, Lau believes it will not be a significant issue, and Allied Control is already working with 3M on establishing industry standards, with potential future integration into ASHRAE (formerly the American Society of Heating, Refrigerating and Air Conditioning Engineers) Liquid Cooling Guidelines.
He said that 3M is also offering a global reverse supply chain for companies which want to dispose of their fluids.
“But if handled properly, these fluids can be still resold to others even after decades, and there is actually an active second-hand market,” Lau said.
Adoption and what’s next
A 'fluid' way to cool high-density data centres Mark Thiele (pic), senior vice president of Data Centre Tech at Switch Communications, believes that immersion cooling has its place, especially in HPC data centres and in homogeneous environments serving a single application, be it cloud or bitcoin mining.
“In five years I would guess maybe a 2-3% market share,” he said when asked about how pervasive immersion cooling systems would be in the market.
Allied Control's Lau noted that two-phase immersion cooling is indeed currently more suited to high performance computing applications paired with high density, and the market is moving towards solutions such as Allied Control’s as it becomes more difficult to cool their systems with other methods.
“Supercomputers were once the sole domain of governments only, but now even supermarkets are using big data analysis to crunch massive amounts of business data to aid critical company decisions and improve bottom-line results, and are therefore buying high density HPC systems,” he said.
Lau said that naturally, any new high performing technology would flourish most in a smaller market first where there is stronger demand.
“It will take some time until the mass market develops in the same direction. You can compare it with how Bill Gates of Microsoft claimed in 1981 that in terms of memory, ‘640KB ought to be enough for anybody’. Technology, in tandem with new demands and expectations, has changed drastically since then.
“But more importantly, the insatiable demand for more processing power is not growing linearly, but rather exponentially, akin to Moore’s Law of the number of transistors doubling approximately every two years. This means that we anticipate the demand for higher density and more effective and efficient datacentre cooling to grow at a similarly staggering rate,” he added.
In terms of hurdles to adoption, Lau admitted that currently cost is still a valid issue, making two-phase immersion cooling only attractive to high-density HPC applications.
The cost stems not from the hardware but rather from the cost of the fluids required.
“One way to mitigate that is to improve the ratio of fluid costs over hardware by designing for a higher density. We expect as well that fluid manufacturers will be able to decrease their costs with improved economies of scale,” said Lau.
He added that the company was also in discussions with a few hardware manufacturers which are intrigued by the possibilities and want to design much denser hardware, which wouldn’t be possible with any other cooling method.
“It will certainly take some time to get adopted by the wider market. Despite that, multiple extremely interested HPC data centre operators have approached us already regarding the purchase of systems, as they face immense problems with other cooling methods at their current densities,” he claimed.
Lau and his team are currently busy working on developing the “next generation” immersion-cooled data centre.
“It will be a containerised system at 1.4MW with the potential to even double to almost 3MW capacity within only 320 sq ft of floor space. This will certainly break all density records on the market by several hundred per cent, and yet be one of the most energy efficient,” he declared.
Lau said that it wasn’t just about making an impact on the industry that fuels the team’s efforts, but rather the larger picture of preserving the environment and improving lives.
“Many Asian countries share the same fate like Hong Kong with the majority of electricity still being produced by burning fossil fuels, so being more energy-efficient will inevitably contribute to lower carbon emissions.
“Also, by reducing space requirements, we hope to especially make a social impact on many crowded Asian cities like Hong Kong with sky-rocketing real estate prices, putting affordability of living space out of reach for many,” he added. 

Related Stories:
Data center issues hampering business strategy
Data centres in Malaysia and Singapore just too cool for words
For more technology news and the latest updates, follow @dnewsasia on Twitter or Like us on Facebook.

Keyword(s) :
Author Name :

By commenting below, you agree to abide by our ground rules.

Subscribe to SNAP
Download Digerati50 2018-2019 PDF

Digerati50 2018-2019

Get and download a digital copy of Digerati50 2018-2019