Thermal Design of Liquid Cooled Microelectronic Equipment

x

PREFACE

With the demands of increased system functionalities and performance, the system power density of microelectronics equipment is continuously increased at a fast pace. This is especially true for the telecommunications systems because the network traffic in the industry has grown very rapidly every year. For the high power air cooled systems, large high performance fans are becoming a must for the systems in order to provide the necessary air flow rates. Two major concerns about these large fans are the power consumption and the acoustic noise of the fans. In addition, the increase in the system power results in a significant increase in the operation cost of the equipment as well as its host facilities such as the date centers. Liquid cooling can not only resolve the above mentioned issues related to high power air-cooled systems but also enhance its system performance and reliability. For some cases, the system power is too high to be cooled by air thermally. The only solution to such situations is adopting the liquid cooling.

Chapter 2 gives the detailed overview of the fundamental principals of convective and boiling heat transfer so that the reader can better understand the physics involved of each subject area. Chapter 3 provides a brief description of the key parts required in a liquid cooling loop. However, the discussion is focusing at the pump which is the heart of any forced convection liquid cooling system. In addition, the discussion is further extended to understanding of function of the flow distribution manifolds. Chapter 4 first gives the detailed discussion about the silicon chip packaging technology and the printed circuit board which provides mounting surface to the chip packages (components) and the wiring channels to serve as conduits for on board chip-to-chip connections. Thermal paths of heat transfer from junctions (heat sources) of the chips to liquid stream and to the printed circuit board are identified. The discussion is further extended to the Gallium Arsenide (GaAs) chips in the microwave modules for the radio frequency (RF) systems. It should be noted that the microwave devices are analog in nature so that the heat generation in the devices equals to the difference between the input and output power of the devices, i.e., the energy loss through the analogy device is in the form of heat.

Chapter 5 provides thorough review and overview of various liquid cooling methods. These liquid cooling technologies include but not limited to natural and forced convection with a single phase liquid or two phase flow, refrigeration cooling, jet impingement and spray cooling. The extensive discussion is also made about microchannel cooling technologies for both a single phase liquid and two phase flow. Chapter 6 examines in details about two phase heat transfer devices which includes heat pipes (HPs), micro heat pipes (μHPs), closed tube and closed

xi

loop thermosyphons, loop heat pipes (LHPs), capillary loop pumps (CLPs), vapor chambers (VCs) and Liquid chambers (LCs). Chapter 7 provides in depth review of currently industry trends of liquid cooled electronics systems including both silicon (Si) devices and gallium arsenide (GaAs) chips in microwave modules. In other words, this chapter provides practical examples how the cooling technologies described in Chapter 5 are being adopted in the functional products available on open market in various industries, especially at electronics and computing systems.

This book places a great deal of emphasis on providing practical solutions to the thermal issues related to high power systems where liquid cooling is required. Chapter 8 provides the thermal design guide for any liquid cooled system. This sets apart from any published books and/or research reports currently available in open literature. This chapter starts with the coolant selection that includes a general selection guideline based on the figure of merit (FOM) of the fluid. To facilitate design and analysis, the most commonly used correlations for the friction factor and heat transfer coefficient with either single phase liquid or two phase flow are summarized in this chapter. The general guidelines for thermal design of liquid cooled systems with focusing at microelectronic equipment are presented. In addition, this chapter provides a step-by-step thermal analysis and design procedure for liquid cooled systems with or without boiling. To meet the needs in telecommunication industry in which no liquid cooled rack is currently available commercially, a detailed system thermal design of liquid cooled telecommunication equipment is made and two thermal design options based on open loop and closed loop cooling schemes, respectively are presented.

Furthermore, the cooling solutions and design procedures discussed here can readily and easily be applied to any systems in other industries.

L.T Yeh

1

Chapter 1

INTRODUCTION

1.1 CHIP/SYSTEM POWER DENSITY

Because of advanced developments in circuit and component technologies, the electric circuit becomes more efficient and thus, heat dissipation from an individual device such as transistor is also less as given in Figure 1.1.[1]. Miniaturization of the circuits greatly decreases the size of individual devices, however, and increases the number of circuits integrated on a single chip. The net result is that the chip heat flux (per unit surface area) has significantly increased in the past decades

There are two popular transistor technologies, namely the bipolar and the complementary metal oxide semiconductor (CMOS). The former is the low impedance device that required relatively high currents to operate resulting in high heat dissipation. On the other hand, the CMOS transistor generally has low power dissipation with high switching speeds.

The needs to increase the chip package density and reduce signal delay time between communicating circuits have led to develop the multichip modules starting in 1970 and is continued as today. The effect of these trends had on module heat flux on the bipolar and CMOS circuit technologies is presented in Figure 1.2 [2]. As can been seen in the figure, the heat flux associated with the bipolar circuit technologies steadily increases from the very beginning and has a big jump in the 1980s. Because of this significant increase in the heat flux, the indirect water cooling systems were introduced in the mid-1980 in order to maintain the chip junction temperatures below their respective limits.

Due to sharp increases of the power on the chips based on the bipolar technologies, it becomes very difficulty to maintain the desired component temperatures and is forced to seek other more energy efficient chip design technologies. One of them is to adopt CMOS based circuit technology in early 1990s which results in significant reduction of chip power dissipation and also makes possible to return to totally air cooled systems again. However, as can be seen in Figure 1.2, the module level heat flux for CMOS based circuit technology for today-systems has also reached to a critical level where new chip design technology may be needed. The most noticeable power increase may be in telecommunications industry. The trend reflected in the growth in the heat flux based upon the footprint area of the equipment is shown in Figure 1.3 [3]. As can be seen from the figure, the

2

system power for the core network is the highest among all products presented in the figure.

FIGURE 1.1 Incredible performance improvement of transistors

FIGURE 1.2 Chronology of module level heat flux in high-end computers

3

FIGURE 1.3 Thermal Loading at Facility for Various Products

Since the current telecommunication systems are still employing the air cooling with exception of a few outdoor systems, large high performance fans are becoming a must for the high power systems in order to provide the necessary air flow rates. In addition to the system thermal issues, two major concerns about these large fans are the power consumption and the acoustic noise of the fans. These concerns can also be applied to non-telecommunications systems.

The COP (coefficient of performance) is often adopted in the telecommunication industry to measure the thermal or energy efficiency of the equipment. The COP is defined as the ratio of the total system power, including power from electronics, power supply/converters, and fans to the total power consumption of fans. Therefore, the higher power consumption of the fan, the lower system COP will be. The COP can also be applied to non-telecommunications systems for measuring thermal or energy efficient of the equipment.

As stated previously, the equipment power consumption is continuously increased at a fast pace. In addition, the cost of powering and cooling in the equipment at the data centers has also increased significantly in the last few years. Therefore, the energy efficiency at the equipment level and the data center become a top priority in the industry.

4

Figure 1.4 [1] illustrates the power consumption in a typical data center. Among them, the power consumption for the IT and cooling equipment are 45.4% and 30%, respectively. Another example for the 1U servers, the ratio of the 3 year site cost to the 1U server cost is given in Figure 1.5 [4]. As can be seen from the figure, the 3-year cost of powering and cooling servers is about 1.5 times of the cost of the servers in 2006 and this factor is projected to be over 22 times at the worst case by 2012. Furthermore, as shown in Figure 1.6 [1], the spend of powering and cooling of IT have greatly exceeded the spending of new servers

FIGURE 1.4 Typical Data Center Energy Breakdown

FIGURE 1.5 Ratio of 3-Year Site Cost to 1U Server Costs

5

FIGURE 1.6 Spending for Power and Cooling versus New servers

1.2 THERMAL TECHNOLOGY MAP

The thermal technology map for various power systems with different cooling schemes is developed and presented in Figure 1.7 [5]. The horizontal axis shows the estimated heat transfer coefficients. The left vertical axis in the figure gives the system power while the right vertical axis represents the system complexity

6

and cost. Generally, the cooling system becomes more complicated and costly when the system power increases. It should be noted that Figure 1.7 should only be used as a general guideline. It is believed that the upper limit of the forced air cooled rack is about 25000 watts. Any system power greater than the above value, liquid cooling may be considered. For practical engineering applications, the forced air cooling is not recommended for any system over 20KW even it works thermally because of acoustic noise and the COP issues.

FIGURE 1.7 Thermal Technology Map

1.3 RELIABILITY AND TEMPERATURE

Reliability that is the reciprocal of the mean time between failures (MTBF) is described as the statistic probability of a device or system operating without failure for a specific time period. Each component or device has its failure rate curve. The reliability of a system is determined by combining many individual part failure rates in series and/or parallel. The system reliability is also affected by the relative importance of the individual part to the system and it can be improved by redundancy at the component, packaging, or system level.

Temperature can also affect the reliability of electronics in many ways, e.g., thermal shock, temperature under continuous operation (steady state), and temperature cycling. In the temperature range specific to electronic equipment, it is an established fact that the reliability of electronics is a strongly inverse function

7

(a near exponential dependency) of a component temperature. The reliability of a silicon chip is decreased by about 10% for every 2°C of temperature rise [6].

FIGURE 1.8 Major Causes of Electronics Failures

The component failure rate is also related to temperature cycling which results from either device power cycling or the cycling of the environment conditions. In a U.S. Navy-sponsored study [6], an eightfold increase in failure rate was encountered in equipment subjected to a deliberate temperature cycling of more than 20°C.

An increased demand on the system performance and reliability also intensifies the needs for better thermal management on electronic equipment. The further evidence of importance of thermal consideration to an electronic system as shown in Figure 1.8 [7] is due to the survey by the U.S. Air Force indicating that more than 50% of all electronics failures are caused by the undesirable temperature control.

The goal of thermal management of electronic system is not only to maintain the temperature of all components below their respective temperature limits but also to achieve as small temperature gradient across the entire system as possible.

1.4 NEEDS FOR LIQUID COOLED SYSTEMS

Air cooling has long been considered and adopted for cooling of electronics. This is especially true for the consumer electronics used in homes and offices. The main reason is that the air cooling is relatively simple, cost effective and safe. However, due to poor thermal properties of the air, there is a system power limitation for the air cooling, especially for the passive cooling of air.

The equipment power consumption is continuously increased at a fast pace. For the high power air cooled systems, large high performance fans are becoming a must for the high power systems in order to provide the necessary air flow rates. Two major concerns about these large fans are the power consumption and the acoustic noise of the fans. The former is related to the equipment energy efficiency while the latter is strictly regulated by the government and industry standards.

The increased system power results in a significant increase in the power consumption and the operation cost of the equipment as well as its host facilities such as the date centers. For some cases, the system power is too high to be considered by air cooling. The only solution to the above situations is adopting the liquid cooling. Even for those high power air cooled systems, the liquid cooling still has following advantages over the air cooled systems:

•   Liquid cooling can support much higher system power

•   Liquid cooling makes equipment much more reliable

–   By eliminating fans, equipment reliability is significantly increased

–   Effective liquid cooling lowers component temperatures

•   Liquid cooled system is much compact and small

–   By eliminating fan tray, air filter, air inlet and exit spaces

–   Slot size (spacing between boards) is small

•   Equipment is almost maintenance free (no need to replace air filter and fans)

–   Operation cost is also reduced by eliminating fans and air filters

8

FIGURE 1.9 Cascade Effect of Energy Savings from Equipment to Data Center

•   Liquid cooled systems can totally eliminate acoustic noise from fans and high speed air flow

•   Eliminating hot air from equipment will reduce data center temperature and decrease energy consumption in data centers

In addition, the power consumption of equipment has a great impact on the power required which is directly related to the total operation cost in host facilities such as data centers. Based on the estimate by Verizon [9], one watt saved on the equipment level will save a total of 2.41 watts in their respective the data center as shown in Figure 1.9. Therefore, it is extremely important for the equipment makers to do a better job on energy efficiency of their products. It should be noted that this cascade factor varies slightly with the individual data centers

REFERENCES

1.   Schmidt, Roger, Thermal Management Applied to Data Centers with Focus on Energy Efficiency, 1st Workshop on Thermal Management in Telecommunication Systems and Data Center, Richardson, Texas, October 25-26, 2010

2.   Chu, R. C, Exploring Innovative Cooling Solutions for IBM's SuperComputing Systems : A Collaborative Trail Blazing Experience, Key-note

9

lecture, 2nd Workshop on Thermal Management in Telecommunication Systems and Data Center, Santa Clara, California, April 25-26, 2012

3.   Brill,K. G., 2000-2010 Product Heat Density Trends Chart, White Paper, The Uptime Institute, 2006

4.   Brill, K. G., Data Center Energy Efficiency and Productivity, White Paper, The Uptime Institute, 2007

5.   Yeh, L.T.,Thermal Management and Energy Efficiency of Telecommunication Equipment, Presented at 2nd Workshop on Thermal Management in Telecommunication Systems and Data Centers, Santa Clara, CA, April 25-26, 2012

6.   Bar-Cohen, A., Kraus, A. D., and Davidson, S.F., Thermal Frontiers in the Design and Packaging of Microelectronic Equipment, Mechanical Engineering, June, 1983.

7.   Hilbert, W.F. and Kube, F.H., Effects on Electronic Equipment Reliability of Temperature Cycling in Equipment, Final Report (Report No. EC-69-400), Grumman Aircraft Corporation, Bethpage, NY, February, 1969.

8.   U.S. Air Force Avionics Integrity Program (AVIP) notes, 1989.

9.  Graff, C., Verizon's Thermal Management Program for Network Equipment Design, 2009

10. Yeh, L. T. and Chu, R. C., Thermal Management of Telecommunications Equipment, ASME Press, 2013

11. Yeh, L. T., Thermal Management of Microelectronic Equipment, 2nd Edition, ASME Press, 2016

10

Chapter 2

FUNDAMENTALS OF

CONVECTIVE AND BOILING

HEAT TRANSFER

In the cooling of electronic equipment, liquid cooling is frequently applied to high power electronic equipment because of its high heat transfer capability. Liquid cooling can further be divided into single- and two-phase flow systems. The latter involves phase change processes such as boiling or condensation that greatly increases the heat transfer capability by utilization of the latent heat. In addition, the phase change process takes place at the constant temperature for a given pressure.

2.1 SINGLE PHASE FORCED CONVECTION

Convection is a transport process involving the combined action of heat conduction and energy storage during fluid mixing motion. Since convection heat transfer is related so closely to the fluid motion. The heat transfer coefficient of a single phase flow under forced convection is a function of the Reynolds number (Re), Prandtl number (Pr), and the thermal boundary conditions as well as the shape of the tube for fully developed laminar flow or for fluids with Prandtl number less than one.

2.1.1 Effect of Reynolds Number

Reynolds number is defined as (ρVL/μ) where ρ is density of fluid, V is velocity, L is the characteristic length, and μ is viscosity of fluid. The Reynolds number also represents the ratio of the inertia force to the friction (or viscous) force. In laminar flow, which corresponds to the case with small Reynolds numbers, there is no mixing in fluid particles, and heat transfer takes place only by molecular conduction due to contact between fluid particles. Since all fluids (with exception of liquid metals) have a small thermal conductivity, the heat transfer coefficients in laminar flow are relatively small. As the Reynolds number (or velocity) increases, the flow regime is changed gradually from the laminar flow to the transition flow which involves a certain amount of mixing between the fluid particles. The mixing motion, even at a small scale, accelerates the heat transfer rate considerably.

11

As the Reynolds number is further increased, the flow becomes fully turbulent except for a very thin layer of fluid immediately adjacent to the wall. In this thin layer which is called the laminar sublayer, the viscous force predominates near the wall surface, and heat transfer takes place mainly by conduction. In the turbulent core, eddies mixing is so efficient that heat is transferred very rapidly between the edge of the laminar sublayer and the turbulent core of the fluid. Although the thermal resistance of the laminar sublayer controls the rate of heat transfer, the overall heat transfer in the turbulent flow is still much higher than that of the laminar flow. This phenomenon is due to the fact that the thickness of the laminar sublayer is very small as compared to the thickness of the entire thermal boundary layer in the laminar flow. It is found that increasing the Reynolds number decreases the thickness of the laminar sublayer and at the same time increases the eddy mixing in the turbulent core. The net result is that the heat transfer rate goes up with the Reynolds number.

2.1.2 Effect of Prandtl Number

The Prandtl number is defined as Pr = ν/α where ν is kinematic viscosity and a is thermal diffusivity of the fluids. The kinematic viscosity, v =μ/ρ, often is referred to as the molecular diffusivity of momentum because it is a measure of the rate of momentum transfer between the molecules. On the other hand, the thermal diffusivity, α = k/ρ cp , of a fluid usually is called the molecular diffusivity of heat, which is a measure of the ratio of the heat transmission and energy storage capacities of the molecules.

The major difference between the air cooling and the liquid cooling is mainly due to the effects of the Prandtl number of the fluids. The Nusselt number or heat transfer coefficient is in general as a function of Pr¹/³. For most of gases, the Prandtl number ranges from 0.7 to 1.0 which basically eliminates the effect of the Prandtl number on the heat transfer coefficient. The Prandtl number also has a great effect on the boundary layer development which can be illustrated in Figure 2.1 where δ and δth are the boundary layer thickness of the hydrodynamic and thermal boundary layers, respectively.

The heat and momentum are diffused through the fluid at the same rate if the Prandtl number is unity (one). If the Prandtl number is greater than one, the velocity profile will develop more rapidly than the temperature profile, whereas, the opposite trends are found for the fluid with the Prandtl number less than one.

It should be noted that the temperature profile starts to develop at the point where the heat transfer between the fluid and wall actually takes place. If the Prandtl number is greater than 5, the velocity profile generally will develop much faster than that of the temperature profile so that a solution based on a fully developed velocity profile will apply quite accurately, even without the hydrodynamic starting length. Such condition is frequently applied to the dielectric fluids that are used widely for electronic equipment because the Prandtl number for most dielectric fluids are, in general, greater than 20 at the room temperatures. On the other hand, for the case of Pr → 0, the velocity profile never actually devel-ops while the temperature profile is developing. A uniform velocity profile (slug flow), therefore, can be assumed over the entire thermal boundary layer.

12

FIGURE 2.1 Prandtl Number Effect on Thermal and Hydraulic Boundary Layers

At very low Prandtl numbers, such as for liquid metals, the thermal resistance is distributed over the entire fluid. The temperature profile for a fluid with a low Prandtl number is similar to that in a laminar flow. In addition, the thermal entry length and the thermal response of the fluid to axial changes in surface temperature are slow and are similar to that of the laminar behavior. For fluids with high Prandtl numbers, very short thermal entry length and quick thermal response to axial surface temperature changes are found. Both are attributed to the con-centration of the thermal resistance in the thin laminar sublayer near the wall and the rapid transfer of heat from the edge of the sublayer over the entire fluid because of effective mixing of fluid particles in the turbulent core.

2.1.3 Effect of Boundary Conditions

In a laminar flow or for fluids with a Prandtl number less than one, the heat transfer coefficient depends on the thermal boundary conditions. This dependency is because the heat transfer mechanism is primarily due to molecular diffusion. Thus, the thermal resistance is distributed over the entire thermal boundary layer thickness, and different boundary conditions or different shapes of the tube cross section simply generate different temperature profiles. For high Prandtl number fluids or turbulent flow, the thermal resistance is mainly in the laminar