Industrial Applications of Large-scale

Fluid-dynamics Simulations
-Expected breakthroughs with large-scale CFD
for Industrial Design-

Outline

Future prospects of applications of CFD to

industrial design

Development and Validations of Flow and

Acoustial Solvers
Development concepts
Code tunings
Validation studies

Application Examples

Future Prospects 2

1
 Future prospects of
applications of CFD to
industrial design

HPC in year 2012 and 2015

K-computer (2012)
88,128 CPUs, 705,024 cores, 10.51 PFLOPS in LINPACK

Worlds top-end supercomputer in 2015

100,000 CPUs, 2 million cores, 100 PFLOPS peak
performance
Peta-flops performance will become available for R&D and design
by direct installation and/or use of HPC clouds in industries. 2012/11/
23 4

2
 Bifurcating Stream in Engineering
CFD
Design optimizations
Tuned and validated RANS (URANS, DES)
Uses up to several hundreds million grids
Corporated with mathematical optimization methods
Requires strong scaling for parallel computations

Numerical prototyping and Investigation of

microscopic flow behavior
Fully-resolved (i.e. wall resolving) LES
Uses billion to trillion grids
Corporated with pre- and post- processing for huge
number of grids
Requires weak scaling for parallel computations
5

Fully-resolved LES (Large Eddy

Simulation)
Resolve such eddies that are responsible to
production of turbulence in TBL
x+=300, y+= 30, z+ =150

Only model eddies in dissipation scale, thus

most reliable

Provide as accurate solution as DNS as long

as flow of interest is properly resolved, with
approximately a 1/100 cost of DNS

3
 How small are the active eddies in
turbulent boundary layer?
Flat-plate turbulent boundary layer at

37 mm 5.1 mm 0.140

25 mm 600 m 0.026

15 mm 77 m 0.005

: Thickness of boundary layer

d: Scale of active eddies in TBL 7

Computer resources required for

fully-resolved LES
Automobile aerodynamics
Free-stream velocity30 m/s
Length scaleL=2 m
Reynolds numberRex=4 million
Friction velocity1.2 m/s
Viscous scale: 12.5 m
Diameter of active smallest vortices0.4 mm
Minimum grid size=0.1 mm (100 m)
Surface grids per 1 m2 100 million
Number of grids in total40 billion
Computational resources40,000-200,000 cores

4
Engineering Applications of
Fully-resolved LES
Applications of LES expected in 2015
products specifications # of TBLs Total # of grids

L=2 m, U=28 m/s (100

automobile
km/h)
4.0 2 40 billion

L= 5m (1/50 scale
model ship model), U=1.0 m/s


D2=300 mm, 1500 rpm,

model pump L=0.15 m, U=24 m/s
12 

D2=40 m, L=0.4 m, U=64

wind turbine
m/s
3 40 billion

D2=600 mm, 1800 rpm,

axial-flow fan
L=0.2 m, U=56 m/s
12 

D2=500 mm, 600 rpm,

propeller fan
L=0.2 m, U=16 m/s
3 

D2=80 mm, 3400 rpm,

small cooling fan L=0.02 m, U=14 m/s
7 

Development and
Validations of Flow and
Acoustial Solvers

10

5
Development Concept

Accurate simulations of turbulent flows and

acoustical fields as to replace tests

Applicable to complex engineering problems

Parallel scalability up to 1 million cores for 1

trillion grid simulations

Easy handling of very large-scale data

11

Governing Equations and

Numerical Methods
Flow Field Simulations (FrontFlow/blue)
LES with Dynamic Smagorinsky Model
Second-order accurate Finite Element Method

Acoustical Field Simulations(FrontFlow/blue-

Acoustics)
Helmholtz Equation solved in frequency domain
with Lighthill tensor computed by LES as source
term

Both codes have been developed, validated

and applied as a part of two National HPC
projects by strong collaboration with AICS
(RIKEN) group 14

6
 Features of FrontFlow/blue
Large data handling:
Input Voxel Mesh
Automatic local grid refiner
Fast-converging matrix solver based on IDR
Weakly-scaled up to unlimited number of cores
Fully-optimized for load/store as well as SIMD
(6.0% sustained for K, 5.0% sustained for BG/P, etc)

User friendliness:
User interface (FrontWorkBench) for handling very
large-scale data
Coupled analyses for aeroaoucstis, fluid-structure
interaction
Various functions: Oversetting, Sliding mesh,
Cavitation model, Free surface flow, etc.

Local grid refinement features

Benchmark test for circular cylinder flow at
Re=100,000
1.5
orignial
refine 1.5 EXP.
1.0 LES (FFB6, No-Refine)
LES (FFB6, Refine)
1.00.5
Cp

0.0
U

0.5
-0.5

0.0-1.0 EXP.
refine
refined LES(FFB6, No-Refine)
LES(FFB6, Refine)
-1.5
-0.5 0 90 180 270 360
-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
Angle [deg]
Y/D

16

7
 Local grid refinement features
Benchmark test for circular cylinder flow at
Re=100,000

1.5 100
EXP.
80 LES(FFB6, No-Refine)
1.0 LES(FFB6, Refine)

60

U ' (% )
U

0.5
40
0.0 EXP.
LES(FFB6, No-Refine) 20
LES(FFB6, Refine)
-0.5 0
-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 -3 -2 -1 0 1 2 3
Y/D Y/D

Time-average velocity Turbulence Fluctuation

17

Hierarchical Voxel Mesh

Voxel Data Interface

18

8
 Benchmark tests for airfoil flow
1 million grids 4 million grids

16 million grids 50 million grids

x=6.2510-3
# of grids=0.6 million

NACA0012 at angle of attack, 9 degrees, Reynolds number of 200,000 19

Benchmark tests for airfoil flow

3.0 0.20
Measured
data01
data02 0.15
2.0 data03
data04
Cd
-Cp

1.0 0.10

Measured
data01
0.0 0.05
data02
data03
data04
-1.0 0.00
0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 15 20
X/C Attack angle / deg.

Surface pressure distribution Drag coefficient

20

9
 Code tunings-1/3
Data reordering for minimizing occurrence of L1, and L2
cache miss

+%45/

Tetrahedral elements Hexahedral elements

21

Code tunings-2/3
Hot kernel performance ratio
Hexahedral element Tetrahedral element

Original (1core) 5.9% 2.4%

Full unroll (1core) 10.8% 4.2%

Full unroll (8core) 5.4% 3.0%

Full unroll
10.2% 10.2%
Reordering (1core)

Full unroll
8.1% 7.7%
Reordering (8core)

22

10
 Code tunings-3/3
Overall Performance ratio and speed-ups

1.E+05

1.E+04

Performance (GFLOPS)
1.E+03

1.E+02

9GFLOPS/
1.E+01
K1Mgrids/node

K1Mgrids/node-*
1.E+00
1.E+00 1.E+01 1.E+02 1.E+03 1.E+04

Num. of Nodes

Ratio of sustained to peak performance Parallel computing performance

23

Flow around a Square Cylinder at

Re=40,000
(number of grids = 50 million)

24

11
Sensitivity Studies on Grid Resolutions

Comparisons of Instantaneous Streamwise Velocity

22 million grid 55 million grid

25

Instantaneous Flow near Cylinder

Comparisons of Instantaneous Streamwise Velocity

Streamwise velocity Streamwise vorticities

26

12
Validation Studies-1/2
Comparisons of Time-averaged Wake Profile
3

U/U0[-]
0

-1
CFD_55Million_Mesh
CFD_22Million_Mesh
-2 CFD_16Million_Mesh
CFD_3Million_Mesh
Sprit_Film_Probe
-3
0 0.5 1 1.5
Y/D

Instantaneous flow Near wake at x=1.5D

27

Validation Studies-2/2
Comparisons of PSD of Streamwise Velocity

Just after separation (5% downstream of corner) 28

13
Validation Studies-2/2
Comparisons of PSD of Streamwise Velocity

Fully developed wake (x=3D) 29

Transitional Boundary Layer on and

Resulting Aeroacoustics from
NACA0012 Aerofoil
(number of grid = 8 million)

30

14
 Prediction of Transitional Boundary
Layer and Resulting Sound
NACA0012: Angle of attack=9 degRe=2.0x105

Vortical Structure near L. E. Short Separation Bubble formed near L. E.

31

Prediction of Transitional Boundary

Layer and Resulting Sound
NACA0012: Angle of attack=9 degRe=2.0x105

Structrues of streamwise vortices Instantaneous distribution of streamwise

vortices on suction surce

32

15
 Time-average and Fluctuating Surface
Pressure Distributions

NACA0012: Angle of attack=9 deg, Re=2.0x105

Surface Pressure Distributions
(NACA0012 at 9deg / Re=2.0e5) 0.8
-4
0.7
-3 0.6

Cp' (RMS)
0.5
-2
0.4
Cp

-1 0.3
0.2
0
0.1
1 0
0 0.2 0.4 0.6 0.8 1
0 0.2 0.4 0.6 0.8 1
x/C
x/C
LES DSM 8M LES DSM 2M Low Res Measured LES DSM8M Measured

Time-average R. M. S.

33

Comparisons of velocity spectra and

far-field sound pressure level
NACA0012: Angle of attack=9 deg, Re=2.0x105

Power Spectral Densities of Velocity Scaled Sound Pressure Levels

(9deg / Suction Side RMS Peak) (NACA0012 at 9deg, Re=2.0E5)
50
1.E+02 Measured

1.E+00 40 LES DSM8M Compact

Scaled-SPL [dB/Hz]

LES DSM8M NonCompact

1.E-02 30
PSD [-]

1.E-04 20
1.E-06
10
1.E-08
0
1.E-10
1 10 100 -10
Frequency [-] 100 1000 10000
Measured LES DSM 8M Slope -5/3 Frequency [Hz]

Velocity fluctuation in short bubble Far-field sound pressure predicted by

acoustics computation

34

16
Tip vortices around NACA0012 Aerofoil
(number of grid = 50 million)

35

Experimental Setup
Measured: drag & lift, surface pressure,
velocity profile, sound

36

17
Instantaneous Flow

37

Validation Studies

Instantaneous flow structures Time-average pressure

distribution near tip(z/c=-0.1)

38

18
 Validation Studies (continued)
PSD of static pressure fluctuations at tip

x/C=0.1 x/C=0.6

39

Predicted far-field sound

Acoustical field computed by Lighthill
equation: w U  a 2 w U
2 2
w2
T
wt 2 w x i2 wxiwx
ij
j

Comparison of sound pressure level Sound field at St=5.7

(760 Hz) 40

19
 Flow and acourtical fields for a model
HVAC

41

Benchmark test for HVAC sound

Exp.ofWakaharaetal.

10MCAAmeshwithmidnodes

10MCAAmeshwithmidnodes,with
penetrationofthesound

Sound field at St=760 Comparison of sound pressure level

42

20
 Application Examples

43

Drag Reduction of Passenger Car by

Controlling Vortical Structure behind Car

(Reynolds number=1.0106,
# of grids = 2 billion)

Collaborator: Toyota Motor Corporation

44

21
How automatic grid refiner works?

Vortices in viscous sub-layer

Overall flow structures

45

TBL and large-scale wake

vortices

Turbulent eddies around an Turbulent eddies

automobile contributing to drag force
46

22
 Comparison with Wind-tunnel
Data
Accuracy Validations

0.3

EXP.

LES

C d[-]
0.2

1.5
1.0 EXP LES
0.5
Cp

0.0
-0.5
-1.0 0.1
-1.5 0.0E+00 2.0E+05 4.0E+05
0.0 1.0 2.0 3.0
X/L R e [-]

Static Pressure Distribution Reynolds Number Dependency

around Car Body of Drag
47

Drag Reduction by Vortex Control

original

Time average RMS

Original shape 0.181 0.0041
Controlled 0.168 (-7%) 0.0031
shape
controlled
Predicted drag coefficients Wake structures
48

23
 Drag prediction for model ship

(Reynolds number=5 million,

# of grids = 8 billion)

Collaborator: SREC

49

Grid-convergence Studies

100 million grids

1 billion grids

8 billion grids

24

Vortex Behaviors

16 million grid Refined 8 billion grid

Quantitative comparisons

Measrued

Computed
Comparisons of velocity profiles Limiting streamlines

25
 Comparisons of Velocity Contours
on Propeller Plane

Hook

Experiment 16 million-grid LES

100 million-grid LES 1 billion-grid LES

T. Nishikawa et al. Application of Fullyresolved Large Eddy Simulation to KVLCC2
Bare Hull Double Model Ship Reynolds Number , Vol. 16 (2013)

Noise Reduction of Small Cooling Fan by

Optimizing Blade/Stator Shapes

(Reynolds number=40,000,
# of grids = 10 milion)

Collaborator: Sanyodenki, Co. Ltd., Fujitsu Ltd.

54

26
Reduction of Turbomachinery
Noise by Mathematical Optimization
Objectives: to develop a very low noise fan by
digital optimizations

Shape
optimizations

Rapid
prototyping

Evaluations of performance
and noise
Rotate at 14,000 rpm

Optimization Loop Static pressure distribution

55

Instantaneous Flow Fields

27
 Validation Studies
Comparisons of SPL with Test Data

Designed flow rate Off-design (partial) flow rate

Low-frequency sound increases by scattering effects!

Integral Equation for Acoustical

Field

Computed Tailored Greens Function for 400 Hz

Incidence field Incidence + scattered field

28
 Reduction of Aerodynamic Noise
by Digital Optimization
Results of Optimizations
Original fan
Noise level
Conventional
fan

Optimized fan
(-6dB)

Optimized fan
Pressure head

Flow rate
Measured Performance and Noise
59

Improvement in Separation Performance

of Cyclone Separator

(Reynolds number=40,000,
# of grids = 17 million)

Collaborator: Konika Minoruta Business Technologies, Inc.

64

29
 Validations of Flow Field LES

U T / | U IN |

0 2 Tangential velocity distributions

Particle Tracking Method

Equations for particle motion

d 2x 24 1
m 2 U air uair  u p (uair  u p ) S p  mg
dt Re 2
Time integration scheme
Runnge-kutta 4

30
 Comparisons of Trapping Efficiency
1.0
Fractional efficiency 

0.8

0.6

0.4
CY70 Re=4.4104

0.2 CY60 Re=2.7104 CY70

0.0
0 5 10 15 20 25
1.0
Volume Ratio

Diameter [m]
130 2000B
Density

0.5
3120
0.0
0 5 10 15 20 25
Particle Diameter: Dp [m]
CY60

Particles Trapped in Longitudinal

Vortices

Tangential velocity distributions

31