Strömungstechnik

Innovative CFD Methoden für Studien zur Wasserstoff-

Sicherheit mit High-Performance Supercomputing (HPSC)
- Weiterentwicklung, Validierung und Performance
Analyse

Werner Rehm

Juli 2001

Kurzfassungsbericht über EU/HGF-CFD/DDT Projekte

Shaker Verlag
 CFD with HPC

The Author: Dr. rer. nat. Werner Rehm, Year of Birth 1942

Physicist in natural science and PhD at RWTH Aachen working as research

scientist, project and work group leader at the Research Centre Jülich (FZJ), who
has gained long-term experience in various aspects of computational fluid
dynamics (CFD) in the aerea of safety research under severe accident conditions,
especially for conventional and nuclear energy systems, including fire prevention
and explosion protection (e.g. HTR-, LWR-, and H2-safety technology). His
current interest is applied computational fluid dynamics (ACFD) with higher
order turbulence and combustion models using high-performance supercomputing
(HPSC) of engineering flows in complex geometries. He is a member of the
ERCOFTAC Pilot Center Germany West, of the CFD-Society in Canada and of
various programme committees. He has performed several international projects
in the field of hydrogen safety technology with relevance in research and practice.
In this context, he has presented many CFD-related publications at conferences
and in journals.

2
 CFD with HPC

Innovative CFD Methoden für Studien zur Wasserstoff-

Sicherheit mit High-Performance Supercomputing (HPSC)
- Weiterentwicklung, Validierung und Performance
Analyse

Kurzfassung

In diesem Report werden die F+E-Arbeitsergebnisse dokumentiert, die im

Rahmen von EU/HGF-Projekttätigkeiten zur numerischen Simulation von
reaktiven Strömungen in komplexen Geometrien durchgeführt werden. Das Ziel
ist die Weiterentwicklung der Methoden zur numerischen Fluiddynamik (CFD),
um mit Hilfe von High-Performance Computing (HPC), die Explosionsprozesse
in technischen Systemen genauer analysieren zu können. Die
Anwendungsbeispiele betreffen konventionelle und nukleare Energiesysteme,
insbesondere die Sicherheitsaspekte einer zukünftigen Wasserstofftechnik. Der
Schwerpunkt liegt bei der Modellierung des störfallbedingten Verhaltens von
Wasserstoff in Sicherheitseinschlüssen hinsichtlich Verteilung und Verbrennung
von Gasgemischen, die bei langsamen, schnellen und überschnellen Flammen
eine Rolle spielen. Für den Brand- und Explosionsschutz werden Modelle und
Kriterien entwickelt, um Übergänge von Deflagrationen in Detonationen (DDT)
sicherheitstechnisch mit adäquaten Gegenmaßnahmen kontrollieren zu können.
Dabei wird das physikalische Mixing-Konzept mit Dilution- und Inertisation
Medien verwendet und vorgeschlagen.

Kurzfassungsbericht über EU/HGF-CFD/DDT Projekte

3
 CFD with HPC

Innovative CFD Methods for Hydrogen Safety Studies

Using High-Performance Supercomputing (HPSC) -
Improvement, Validation and Performance Analysis

Abstract

This report describes the R&D work performed within the scope of
EC/HGF project activities concerning the numerical simulation of reacting flow in
complex geometries. The aim is the refinement of numerical methods used in
computational fluid dynamics (CFD) by introducing high-performance
computations (HPC) to analyse explosion processes in technical systems in more
detail. Application examples concern conventional and nuclear energy systems,
especially the safety aspects of future hydrogen technology.
The project work is mainly focused on the modelling of the accident-
related behaviour of hydrogen in safety enclosures regarding the distribution and
combustion of burnable gas mixtures, ranging from slow to fast or even rapid
flames. For fire and explosion protection, special models and criteria are being
developed for the assessment of adequate safety measures to control deflagration-
to-detonation transition (DDT) processes. Therefore, the physical mixing concept
with dilution and inertization media is applied and recommended.

Short Summary Report of EC/HGF-CFD/DDT Projects

4
 CFD with HPC

Contents of the Report

Preface .............................................................................................................. 6

Overview ........................................................................................................... 8

Synopsis ............................................................................................................ 9

Technical Summary ......................................................................................... 11

Conclusion and Outlook .................................................................................. 42

References ....................................................................................................... 44

Acknowledgments ........................................................................................... 47

Annex: Field Codes and Supercomputing ......................................................... 48

Nomenclature .................................................................................................. 61

Curriculum Vitae ............................................................................................. 63

5
 CFD with HPC

Preface

The assessment of the integrity of safety enclosures under severe accident

conditions is an important safety issue for conventional and nuclear power plants,
especially safety studies for severe accident conditions exploring innovative
safety trends. These studies are under consideration to improve the safety
behaviour of technical systems and to reduce the consequences of accidents for
the environment as far as possible. To limit the increasing global climate problem
and the greenhouse effect, nuclear power and hydrogen technology are potential
future energy options. However, increasing safety requirements have to be
fulfilled to minimize industrial risks and to harmonize the safety culture.
In the case of severe accidents in water-cooled nuclear reactors, for
instance, significant quantities of hydrogen and steam could be released into the
atmosphere of the containment, locally reaching flammability or even detonability
limits. To control accidental hydrogen combustion, different safety measures have
been developed as part of the defence-in-depth concept with several accident
management strategies. However, concerning flame acceleration or hot jet mecha-
nisms, for example, a transition from deflagration to detonation (DDT) cannot be
excluded a priori and may produce very high explosion loads jeopardizing safety
equipment and enclosure systems.
In this context, several research projects based on computational fluid
dynamics (CFD) have been established to study the effects of DDT on hydrogen
explosion loads in large-scale enclosures. These joint projects involved a close
collaboration between research centers, universities, and industry, e.g. FZJ, FZK,
FUB, RWTH, AEA, CDL, INCAS, and UPM, with know-how transfer leading to
synergy and spin-off effects. The R&D effort was supported by the HGF senate

6
 CFD with HPC

in Bonn and the EC directorate in Brussels within the scope of joint hydrogen
safety projects based on an international cooperation.
This productive cooperation was focused on hydrogen distribution,
combustion, flame acceleration and DDT processes, including CFD code impro-
vement with validation and the exploration of mitigation or prevention features.
Main aspects are related to scientific high-performance supercomputing (HPSC)
for large-scale simulations of reactive flows in complex geometries, using vector
and parallel processing capabilities, including aspects of fire protection and
explosion prevention for innovative energy systems, especially for nuclear safety
and hydrogen safety with relevance to industrial safety. Besides the optimization
of modern CFD methods with supercomputing for safety analyses, further studies
were performed for hydrogen control systems, which are used in conventional fire
protection and industrial explosion prevention related to process and
environmental safety. Therefore, several industrial companies were contacted,
e.g. ESMG, INBUREX, FIKE, BINDER, TOTAL WALTHER, KIDDE-
DEUGRA, and HYDROGAS.
In this context, we also would like to thank foreign partners from Europe,
Canada, USA, and Japan for their helpful discussions in the field of fluid dynamic
safety analyses and hydrogen safety concepts for nuclear and conventional
applications relevant in research and practice.

Werner Rehm, July 2001

Project Leader and Coordinator c/o
FZJ Research Centre Jülich, Germany
ISR Institute for Safety Research and Reactor Technology
Email: w.rehm@fz-juelich.de or w.rehm@t-online.de
http://www.fz-juelich.de/isr/
http://isr230.isr.kfa-juelich.de

7
 CFD with HPC

Overview

The final documentation of the HGF-H2 Project in FZJ report Part 2: CFD
Software Optimization and Special Application consists of the following
reports:
1. Final Report Part 1: Advanced CFD methods with supercomputing -
improvement, validation, and performance, by W. Rehm, C. Nae, et al.,
Aug. 2001, FZJ Report
2. Detailed Report Part 2: Recent DDT simulations and criteria based on
experimental and numerical results, by B. Wang, W. Rehm, Sep. 2001, FZJ
Report
3. Analysis Report Part 3: Benchmark calculations and performance analysis
for special application examples, by W. Jahn, W. Rehm, Oct. 2001, FZJ
Report
4. Summary Report Part 4: Evaluation and application plan for fluid
dynamic safety analysis, by W. Rehm, W. Jahn, C. Nae, B. Wang, Nov.
2001, FZJ Report

Foundation: The basis of the final documentaion of Part 2 is the EC project

on H2DDT and the further development required within the scope of the HGF-H2
Project based on the project proposal and the positive vote of the HGF reviewer
for the FZJ’s work programme outlined in the presentation for the evaluation.
During the working period between 1998 and 2001, both EC and HGF projects
were performed in parallel and have been documented and the results published
in several progress reports and presented at international conferences. In the final
report Part 1, we present in selected chapters the main project work and major
results that have been obtained together with our project partners, s. Ref. [27].

8
 CFD with HPC

Synopsis

In this short summary report, we present the R&D work that is being
performed within the framework of applied computational fluid dynamics
(ACFD) with high-performance supercomputing clusters (HPSC) for joint
EC/HGF-funded CFD/DDT projects, aimed at the numerical simulation of
turbulent reacting flows in complex geometries at the so-called GGG computing
level to gain high numerical resolution in space and time. As a result, it is shown
that scientific-technical high-performance supercomputing networking (HPSN)
has the capability to enhance a close interdisciplinary collaboration between
universities, research centres, and industry. Major aspects are related to the
prediction of physical-chemical phenomena using a modern field code cluster
(MFCC), solving the reactive Navier-Stokes/Euler equations with higher order
turbulence and combustion models on a powerful platform, including code
verification and validation of numerical results by experimental observation.
In this context, special CFD project results are outlined in five selected
chapters relating to hydrogen behaviour - distribution and combustion - as well as
the transition from deflagration to detonation (DDT) in explosive hydrogen
mixtures using CRAY supercomputing systems and different parallelization
strategies. The report focuses on CFD optimization and application studies. The
main task is to obtain a sufficiently high numerical resolution of reactive flows in
complex geometries using supercomputing to study fire and explosion loads. In
addition, it is shown in the appendix that each grid size has an optimum number
of processor nodes for parallel processing. This was analysed in more detail for
large-scale multi-dimensional simulations of non/reacting flows based on several
distribution and combustion test cases. Furthermore, these project studies are
relevant with regard to the maintenance of scientific competence and expertise at

9
 CFD with HPC

FZ Jülich to improve the safety culture for innovative energy systems by

combining basic research work and end user practice for the industrial safety of
technical systems, including fire protection and explosion prevention in
conventional and nuclear power plants, especially for hydrogen technology as a
future energy option. Besides the modern CFD methodology developed with
supercomputing, a major effort was made in the field of industrial safety to
control fire and explosions inside and outside of safety enclosures, studying the
various safety techniques for process safety and industrial explosion protection,
including the state of the art and new development directions with federal
regulation trends.
The H2-CFD/DDT-related research work has been performed during the
past few years at the Institute for Safety Research and Reactor Technology in the
ISR1 Department of Research Centre Jülich (FZJ). We gratefully acknowledge
the ISR directory with the critical guideline and we appreciate the scientific
reorganization of the ISR, which was initiated by the board of the directors. The
author is grateful to all partners for supporting and funding this project work,
especially the HGF senate in Bonn and the EC directorate in Brussels for their
valuable consultations. The EU-H2DDT project work was successfully
completed with seven European partners and is documented in FZJ-ISR Report
Vol. 9, reaching extensive consensus. The following German HGF project on H2
safety for nuclear reactors was performed by FZ Karlsruhe and FZ Jülich
consisting of Part 1, Part 2, and Part 3. In Part 2 we obtained recent results with
our partners which are different from Part 1 and Part 3, apart from a harmonized
compromise in the controversial debate about hydrogen safety features.

10
 CFD with HPC

Technical Summary

Safety Aspects: Generally, fire and explosion protection is an important

safety issue in most areas of engineering. For instance, the prediction of hydrogen
explosion loads in safety enclosures (reinforced concrete containment or steel
vessel systems) under severe accident conditions is relevant for nuclear reactor
containments and hydrogen demonstration plants with fuel cells or hydrogen-
powered vehicles. Whether or not a transition from deflagration to detonation
(DDT) could occur in safety enclosures depends on the specific plant design and
the anticipated accident scenario. Explosion phenomena are under investigation
worldwide due to the high danger potential, as experimental observation has
shown. The DDT phenomena have been extensively studied so that most of them
are qualitatively understood. However, quantitative methods have not yet been
established for objective assessment in large-scale enclosures. DDT is a very
complex process associated with turbulent reacting flows and shock wave
interactions where rigorous closure solutions are still in the distant future so that
practical approaches are needed. The major problems are related to the different
time and length scales of the physical-chemical processes involved, which are
difficult to predict in realistic safety geometries for ambient conditions. It is well
known that the likelihood of DDT increases with the geometrical scale and that
for the numerical treatment huge computer resources are necessary to gain a
sufficiently high numerical resolution both in space and time. (Note: It is an
ambitious task to define detonation limits in lean hydrogen-air mixtures for large-
scale geometries). Therefore, it is reasonable to subdivide the potential DDT
processes into essential stages, e.g. for slow, fast, and rapid flames in the
sub/super-sonic regime, and to develop suitable software tools capable of
numerically defining DDT conditions or criteria to control the resulting
combustion mode and the load response.
11
 CFD with HPC

Objectives and Aims: The HGF project on reactor safety addresses

innovative methods of the analysis and control of hydrogen behaviour in core
meltdown accidents. The project consists of three parts. Part 2 of the project is
performed by FZJ-ISR1 and is related to: CFD Software Optimization with
Nuclear and Non-nuclear (Special) Applications. The prediction of hydrogen
behaviour under severe accident conditions is an important safety concern for the
energy systems of nuclear and conventional power plants to accommodate
increasing safety regulations by advanced safety methods thus minimizing the
hydrogen risk for industrial safety.
In general, hydrogen behaviour consists of two classical phenomena, namely
hydrogen distribution and hydrogen combustion. Both phenomena are associated
with turbulent flows. The description of turbulent flows is one of the most
difficult problems to solve in continuum physics. Depending on the information
needed for the details of the flow field, different physical models and numerical
techniques have been developed in the literature to solve the continuum equations
with the conservation quantities in the form of so-called Navier-Stokes equations.
Therefore, the prediction of hydrogen behaviour has, in principle, two main
aspects. The first one is a more scientific aspect pursued in the research work of
universities and the second one is a more engineering aspect focused on the end-
user work of industry. For this reason, in the past few years a wide range of
predictive tools have been developed for various application aims, ranging from
lumped parameter codes to multi-dimensional field codes. Following the project
aim of developing innovative methods for the prediction of hydrogen behaviour,
the underlying basic idea of Part 2 was, as a working hypothesis, to combine
selected computational fluid dynamics (CFD) tools in a modern field code cluster
(MFCC) with supercomputing capabilities, which are useful and suitable for
research work as well as for end-user practice, representing the state of the art of

12
 CFD with HPC

non-reacting flow simulations (distribution) and reacting flow simulations

(combustion) in complex geometries.
The selection and evaluation of the basic CFD codes used at FZJ in Part 2
was made by expert consulting, professional assistance from academics or
commercial companies, and a comprehensive literature review, e.g. Ref. [1-17],
based on our own experience already gained in the EU-H2DDT project to fulfil
the tasks outlined in the project proposal and proven by the positive vote of the
HGF reviewer committee. The main tasks include: improvement, validation,
performance and optimization of numerical simulations with supercomputing
systems for non-reacting flows (turbulent fluid), reacting flows (turbulent
combustion), including transition from deflagration to detonation (DDT)
regarding hydrogen safety analyses with fire and explosion protection for
technical systems. Proof tests and test cases are more closely related to non-
nuclear applications (hydrogen technology) and less to nuclear applications
(reactor technology), performing special single effect studies from a generic
safety point of view on the basis of the specific R&D programme of FZ Jülich.

Work Tasks: Within the scope of Part 2, special subtasks were planned in
more detail and described in the project proposal with the following topics :
• Improvement, optimization, and application of modern field codes for Fluid
Dynamic Safety Analysis in the FZJ for technical systems, especially for
hydrogen safety.
• A probabilistic safety analysis (PSA) was not performed since the working
group moved to a different department. However, PSA studies were
performed by NNC Knutsford as a partner in the EU-H2DDT project so that
these results are also available.
• A safety analysis of structural mechanics could not be performed because of
the closing period of the working group. But the finite element program

13
 CFD with HPC

DYNA was ported, parallelized and tested on the T3E in a separate FZJ
project and is now also ready to use.

As a consequence, the project work of Part 2 was focused on Applied

Computational Fluid Dynamics (ACFD) with High-Performance Super-
Computing (HPSC) for the simulation of reactive flows in complex geometries,
including multi-fluid flows, particle transport and fluid-structure interactions. In
principle, the project team was able to determine the basic items and issues,
which were outlined in the HGF-H2 work programme and specified in the FZJ
tasks for detailed CFD/DDT studies. Concerning the H2 analysis, the R&D work
was mostly performed for dry hydrogen-air mixtures as worst case
considerations. With respect to H2 control, the mixing concept was selected with
dilution/inertization media as suppression agents. Considerable progress was thus
made in analysing hydrogen safety conditions and defining safety criteria for
innovative power plants as a future energy option, which are useful for limiting
the increasing climate problem and fulfilling advanced safety requirements
minimizing industrial risks.

Contents: The final report consists of five selected chapters. The first
chapter outlines applied computational fluid dynamics (ACFD) using high-
performance supercomputer clusters (HPSC). The second chapter describes
large-scale simulations for complex engineering flows with recent solvers,
including LES methods. The third chapter summarizes experimental and
numerical studies of explosion phenomena in hydrogen-air mixtures under severe
accident conditions. The fourth chapter highlights reactive flow simulations in
complex geometries with high-performance supercomputing. The fifth chapter
gives an overview of application examples for hydrogen safety technology,
including safety systems for fire protection and explosion prevention used in

14
 CFD with HPC

process and industrial safety. Besides the conclusion with outlook, an extensive
bibliography has also been compiled. Finally, the appendix collects field code
equations, the main results of parallel processing with a large number of
performance and analysis tests, including representative input data for some CFD
codes. Great effort was involved in preparing the input data sets for the several
field codes used in various test cases, including the handling of input/output data
for pre- and post-processing with visualization or animation.

Basic Equations and Field Codes: The fluid motion can be described by
field codes solving the conservation equations for mass, momentum, and energy,
which form together with the equation of state a closed mathematical system, i.e.
the so-called Navier-Stokes (Na-St.) equations. The governing equations are
formulated, as follows:

(1) ∂ρ / ∂t + div (ρ u) = 0;
(2) ∂(ρ uj) / ∂t + div(ρ uj u) = - grad p + div(µ grad uj) + SM ;
(3) ∂(ρ i) / ∂t + div(ρ i u) = - p div u + div(k grad T) + Si ;
(4) p = p (ρ, T) and i = i (ρ, T).

These governing equations for rate of change + convection - diffusion =

source term can be expressed for a general variable (Φ) in the generic form, as to:
∂ (ρ Φ) ⁄ ∂ t + ∇ (ρ Φ u - Γ ∇ Φ) = S Φ

with the closure terms:

(a) Standard turbulence- and combustion models are based on the eddy
viscosity (µ) and a two-equation turbulence model (k-eps): µ T = C µ ρ k 2 / ε with
the eddy life time τ e = k / ε.

15
 CFD with HPC

(b) A turbulent combustion model (S T ): S T = - A T ρ ( ε/k ) m lim and the

Damköhler number being larger than unity (Da = τe /τch > 1).
(c) A shock compression heating (S C ) with a reaction model: S C = A c ( ρ / τ

ch ) for T > T ig using the chemical induction time (τ ch) and the ignition
temperature (T ig.).
The basic partial differntial equations (PDEs) are integrated in a computing
domain, using the conservative form of finite volume methods (FVM) with
various accurate differencing schemes and robust solver techniques for the
convection, diffusion and source terms with ambient initial and boundary
conditions. The resulting set of algebraic equations is solved with direct or
indirect algorithms (i.e. iterative or coupled solver), as to: a(p) Q(p) = a(w) Q(w)
+ a(e) Q(e) for the residual reduction to the defined values and aimed at grid-
independent solutions. Here, we select from verification of codes versus
validation of results, as to an independent end-user point of view.
The basic equations contain all of the physics of a given turbulent flow, due
to the fact that turbulence is a continuum phenomenon. Turbulence modelling is
one of three key elements in CFD using RANS, LES or DNS methods, (Fig. 1).
Mathematical theories have been developed for the other two key elements, i.e.
grid generation and algorithm development. An other issue is related to the
combustion modelling of premixed flames, (Fig.2).
For the numerical simulation of transient, compressible, turbulent, chemi-
cally reacting flows, we have developed special field codes for specific studies
providing new versions of reactive Navier-Stokes and on reactive Euler flow sol-
ver codes. The software and hardware systems of the modern field code cluster
are based on an innovative methodology for CFD predictions of engineering
flows in complex geometries using high-performance supercomputing facilities.

16
 CFD with HPC

Present Flow Solvers and Combustion Models

Computing Effort

Research Development Standard

high DNS
DNS
LES
LES
Resolution

Subgrid Scale Models VLES

VLES
RNS
Mean Values RANS
RNS
low
Energy Spectrum

small Eddies large Eddies resolved

Kolmogorov Integral Scales

Fig. 1: Overview on Navier-Stokes flow solver methods with DNS, LES, VLES,
and RANS techniques used in research, development and standard applications.

Mixed-is-Burnt Model
PDF Model DDT

Combustor Eddy
Dissipation
Model Engine

Flamelet Model

lam. gewinkelte Flammen

Fig. 2 Combustion models for premixed flames as to the Borghi diagram, i.e.
flame velocity ratio versus flame length scale ratio with the parameters
Damköhler and Karlovitz numbers.

17
 CFD with HPC

Modern Software and Hardware Infrastructure

Deflagration PVP System MPP System Detonation

CFX-PVM T90 T3E-600 T3E-1200 DET-HPF
Reactive 10 Proc. 512 Proc. 512 Proc. Reactive
Navier-Stokes 8 GBytes 64 GBytes 262 GBytes Euler
18 GFLOPS 300 GFLOPS 614 GFLOPS
Solver Solver

MFCC-MPP: D3UNS, CFX, AIXCO, DET, (COM, DYNA)

DEC-WS J90 ERCO-DDT

Pre- and Post- HPSC-
SHOCKIN 16 Proc. Semi-Implicit
Processing 8 GBytes Flame-Shock
IFSAS Networking
BUILD, FIEDVIEW 3 GFLOPS Na-St. Solver

Fluid Structure Dynamics File Server Transition

Fig. 3: Modern field code cluster (MFCC) with high-performance super-

computing on the CRAY computer complex at FZJ for the simulation of reactive
flows in complex geometries with parallel processing.

Tab. 1: Main CFD/DDT-related computer models used in hydrogen safety.

• Distribution Mode (Concentration): Non-reactive Na-St. Solver D2UNS-MPI (FZJ)
Unstructured Navier-Stokes Flow Solver with LES Modelling
• Deflagration Mode (Flame Acceleration): Reactive Na-St Solver CFX-3D/MPI
(AEA) Higher Order Turbulence and Combustion Models with Unstructured Grids
• DDT Mode (Auto-Ignition): Reactive Na-St Solver AIXCO-2D/MPI (RWTH)
Flame Front Tracking/Shock Wave Capturing Schemes and Reduced Reactions
• DDT Mode (Shock-Ignition): Reactive Euler Solver SHOCKIN-2D/AG (FZJ)
Unstructured Reactive Euler Flow Solver with Reduced and Detailed Chemistry
• Detonation Mode (Propagation): Reactive Euler Solver DET-2D/HPF (FZK)
Hydrodynamics with Chemical Source Term and One-Step Reaction Kinetics
• Fluid-Structure Analysis (Loads): Reactive Euler Solver IFSAS-3D/AG (CDL)
Shock/Detonation Ignition Models with Adaptive Grids for Explosion Loads
• Structure Mechanics (Response): Finite Element Code DYNA-3D/PVM for static
and dynamic deformations of safety enclosures (CONDAT)

18
 CFD with HPC

Innovative CFD Methodology for Hydrogen Safety

Slow Flame Fast Flame Rapid Flame

Deflagration Explosion/DDT Detonation

Modelling Phenomena

Flame AICC
Slow Flame
Scenario
Codes subsonic Pressure

Deflagration Acceleration

CFX
Fast Flame Shock
AIXCO
turbulent, sonic Pressure
SHOCKIN

Explosion Transition DDT

DET Rapid Flame CJ

IFSAS supersonic Pressure

Detonation Propagation

IFSAS Combustion Loads Hazard

DYNA Pressure, Impulse
Potential

Prediction of Explosion Loads in Safety Enclosures

Hydrogen Flames and Modern CFD Codes Using Supercomputing
for the Prediction of Explosion Loads in Complex Geometries

Figs. 4: Hydrogen flame scenario (above) and reactive field codes for hydrogen
safety technology (below) of technical energy systems.

19
 CFD with HPC

Results: In Part 2 of the HGF-H2 project, we developed modern CFD

methods for the analysis of hydrogen combustion studies using supercomputing
capabilities related to software optimization and special applications including
improvement, validation, and performance tests. The modern field code cluster
(MFCC) developed consists of several reactive Navier-Stokes/Euler flow solvers:
D2UNS, CFX, AIXCO, SHOCKIN, DET, and IFSAS with an advanced platform
for pre- and post-processing performed with the PATRAN, BUILD and AVS,
FIELDVIEW tools on fast workstations. Characteristics are hybrid grids, higher
order turbulence and combustion models with advanced solver techniques,
including supercomputing with parallel vector processing (PVP) and massively
parallel processing (MPP), which is executed on the CRAY supercomputer
complex (J90, T90, and T3E) of the FZJ, (Fig. 3 and Tab. 1). Our modern field
code cluster is capable of numerically simulating non/reacting flows of slow, fast,
and rapid hydrogen flames in complex geometries with resulting loads, (Fig. 4),
and consists of the following powerful software and hardware infrastructure:

• D2UNS Code: We developed a new unstructured Navier-Stokes solver

for non-reacting flow simulations (hydrogen distribution) consisting of several
classical turbulence modelling and large eddy simulations (LES) for compressible
flows, including subgrid scale (SGS) modelling based on a Smagorinsky model
with wall damping, wall laws and dynamic modelling by Germano. The code
works quite well on the T3E in the MPP mode and was nowadays successfully
tested for benchmark test cases reported in the literature (e.g. channel flow
around obstacles). Presently, 3-dimensional versions are under development
together with multi-fluid models and options for reacting source terms. The code
is a sophisticated research tool for the large-scale simulation of engineering
flows, especially for turbulent mixing with inertization, making full use of mesh

20
 CFD with HPC

refinement and adaptation using flow field parameters. Special information is

highlighted in the annex.

• SHOCKIN Code: For the numerical simulation of reacting flows

related to hydrogen (auto)self-ignition (i.e. spontaneous thermal ignition), we
developed a new adaptive reactive Euler solver with a two-step reaction model,
including reduced or detailed chemistry for the influence of active/passive
additives. The code was validated for various H2-DDT experiments performed at
SWL of RWTH Aachen showing an excellent agreement between experimental
and numerical results and providing detailed insights into shock wave behaviour.
For the first time, H2-DDT test simulations were extensively performed for self-
ignition conditions in a large-scale containment model geometry with about
50.000 m**3 volume. The code is an advanced research tool for analysing DDT
self-ignition conditions for complex geometries and for various hydrogen-air
mixtures, including dilution and inertization effects to prevent explosions.
Specific information is outlined in the annex.

• CFX Code: For the prediction of non/reacting flows (hydrogen

distribution and combustion) in un/confined geometries, we significantly
improved the CFX software. This software consists of CFX-4 version (structured
grids) and CFX-5 version (unstructured grids) with body-fitted coordinates. CFX-
4 includes a variety of physical-chemical models, whereas CFX-5 was extended
for new wall functions and eddy dissipation combustion models (EDM) with
flame-quenching modelling. Various classical turbulence models can be applied
(e.g. k-epsilon, k-omega, RNG, etc.). We implemented a new LES model in a
first test version for in/compressible flows based on a Smagorinsky model with
modified wall functions for non-reacting flows, which was tested for tube and

21
 CFD with HPC

channel flows with obstacles. Throughout this approach, the subgrid viscosity is
computed by the subgrid scale model ( ν s ) which can be written in the form of:

ν s = (c s D ∆ ) 2 ( 2 S ij S ij ) 1/2

with the S ij resolved strain rate tensor using ∆ the grid cell, D the wall daming,
and c s as model constant. The CFX-4 version was ported to the T90 for the PVP
mode and CFX-5 was optimized on the T3E for the MPP mode. Performance
tests showed a good scalability with increasing grid cells and number of
processor nodes. Both new versions of CFX-4/5 were extensively used and
validated in experiments for hydrogen distribution and combustion tests,
especially for flame acceleration with repeated obstacles (RUT tests), for hot jet
ignition in multi-room tests (BMC tests), and for flame-quenching tests (ENEL
test). Empirical parameters were fine tuned for the selected test cases indicating
that the experimental results agreed quite well with the numerical simulations.
The CFX-4/5 code system is a comprehensive design and analysis tool for
numerical simulations of industrial flows including fire and explosion protection.
For instance, Fig. 5 displays the numerical simulation of a turbulent H2-flame
acceleration in the channel with repeated obstacles into the cavity of the large-
scale explosion RUT facility with about 500 m**3 volume.

• AIXCO Code: For the numerical simulation of hydrogen combustion

(flame acceleration and DDT processes), we essentially improved the reactive
Navier-Stokes code AIXCO. This code is based on the flamelet combustion
model with flame front tracking and shock wave capturing schemes. The flame
front is described as a discontinuity between the burnt and unburnt gas, using a
level set formulation ( G equation ) with appropriate burning laws ( s ) which are

22
 CFD with HPC

derived from combustion theory or experimental results. The associated level set
function evolution equation can be written, as follows:

G t + ( v + s n ) ∇ G = 0; and | ∇ G | = 1

with G ( x, t ) = flame front level set, s = normal burning velocity, and v = flow
velocity. The effects of turbulence can be taken into account through classical
two-equation models (e.g. k-eps.). In addition, a two-step reaction model is
implemented to study self-ignition characteristics of various hydrogen-air
mixtures based on reduced chemical reaction schemes.
Furthermore, the code is able to simulate the transition from a deflagration
wave to a detonation wave, using a switching criterion defined by the critical
ignition temperature. We ported, parallelized and optimized the AIXCO code on
the CRAY-T3E, obtaining good performance tests and reasonable speedup
factors with increasing grid cells and processor nodes up to 500 processors (600
Mhz). The code has now been tested in small-scale laboratory DDT tests
provided by SWL at RWTH Aachen and large-scale explosion tests performed in
RUT-KI, Moscow, reviewing the DDT modelling. Finally, the code was extended
for very large eddy simulation (VLES) using two-scale turbulence modelling with
down- and up-scaling between the integral and the subgrid scales for non-reacting
flows, e.g. mixing layers, jets, wakes, and channel flows with obstacles.
Recently, the VLES turbulence model is adapted in combination with the flamelet
combustion model for the numerical simulation of turbulent reacting flows. In
conclusion, the AIXCO code is a modern research tool for the prediction of
turbulent combustion, based on RANS, VLES, or DNS flow solver options used
for combustion engines and hydrogen safety. For example, Fig, 6 demonstrates
the numerical simulation of a H2-flame interaction with shock wave focusing in
the corner of the RUT test cavity reaching auto-ignition conditions in the unburnt
gas ahead of the flame with DDT pre-conditioning.

23
 CFD with HPC

Temperature Field: 300 to 1600 K Fast Flame

Pressure Field: 1 to 5 bar Shock Focusing

Fig. 5: CFX (k-e, EDM) test example for flame acceleration (FA) in a channel
with temperature and pressure contours (11% H2-air at NTP).

burnt gas

unburnt gas

Density (kg/m**3)
Time = 0.178 s

Fig. 6: AIXCO (k-e, FC) test case for DDT flame-shock wave interaction in a
cavity with density contour (15% H2-air at NTP).

in out

Cavity
DDT Formation of Detonation at 20 ms

Pressure (bar)
0.3 36

Fig.7: IFSAS (1-step, AG) test case for shock wave focusing in a corner with
DDT self-ignition and detonation formation in an explosive hydrogen-air
mixture as to pressure contour (15% H2-air at NTP).

24
 CFD with HPC

• IFSAS Code: For the detailed analysis of explosion phenomena with

relevance to industrial safety, we implemented a new version of the Integrated
Fluid Structure Analysis Software (IFSAS) with the SuperSTATE module
concerning shock ignition, detonation propagation, multi-phase flows, dust
explosion, fluid-structure analysis, and load response. For resolved computations
and for detonation wave simulation a standard Arrhenius law is normally used in
the fuel consumption process as described:

( ρ f ) t + ∇ ( ρ v f ) = - ρ f b exp ( - E / RT ) with T = p / ρ R

or a more sophisticated chemical rate law can be employed. The code uses
adaptive grids for static and dynamic mesh refinement, including slanted
boundaries. The code was tested and validated for representative explosion
processes (SWL/RUT tests), especially for single-effect studies associated with
shock ignition, detonation propagation over obstacles with resulting loadings in
confined and partly confined geometries for various explosive mixtures and
ambient conditions. For practical applications, an extended version is under
development. Indeed, a typical test case is presented in Fig. 7 showing the
numerical simulation of shock wave focusing with self-ignition and detonation
formation of an explosive H2-air mixture in the RUT test cavity with high DDT
pressure peaks.

• Supercomputing: To analyse the different aspects related to hydrogen

safety, these special CFD/DDT codes were developed for specific studies with a
high numerical resolution. We ported the modules of the modern field code
cluster (MFCC) on several computer platforms, ranging from fast workstations to
powerful supercomputers, with respect to meta-computing to increase the
available computer resources, depending on the required details of the

25
 CFD with HPC

non/reacting flow field. Most of the modules of the MFCC software system were
optimized for parallel processing on the CRAY supercomputer complex at FZJ as
a center of excellence. The CRAY complex consists of the J90, T90, and T3E,
which are integrated in a high-speed network operating in the upper GGG-
computing class, whereas the teraflops class is already under investigation for
even more challenging tasks. In this context, we studied several parallelization
strategies, focusing on massively parallel processing (MPP) using multi-blocks,
domain decomposition and message passing tools (HPF, PVM, and MPI). The
performance results were compared with the theoretical values for linear speeding
(Sp) defined by the Amdahl‘s law, as to:

Sp (a) = p / (1 + (p - 1) a ) with p = no. of processors and a = sequential part.

We will therefore use results obtained in other projects, such as UNICORE

(access), KOJCEK (performance), GIGABIT (test bed west), for high-speed
networking and clustering in Europe based on EC/OECD-initiated project plans.
Recently, we peform large-scale simulations of the order of some million grid
points with about 500 processor nodes obtaining a reasonable performance,
scalability, and efficiency above 50 %, especially for large eddy simulations
(LES) of engineering flows for more challenging tasks, (Fig. 8).

• H2 Safety: Hydrogen control systems are based on mitigation,

prevention, or protection measures in various technical applications. For fire
protection and explosion prevention, effective suppression agents have been
studied recently and already being used in industrial safety, public buildings, and
high-tech equipment (Fig. 9). The effectiveness depends on the special plant
design and the specific requirements, which are under investigation in further
project activities with relevance to industrial safety, Ref. [14].

26
 CFD with HPC

Fig. 8: Speedup for massively parallel processing on the CRAY-T3E with the
CFX-LES test case in channel flows over obstacles using 3.000.000 grid points.

Fig. 9: Fire protection system with FM-200 spreeding in the control tower of the
airport at Düsseldorf by KD.

27
 CFD with HPC

Collaboration: The project work of Part 2 was performed in close

collabaration with the ITM of RWTH in Aachen concerning the further
development of the flamelet combustion code AIXCO code, especially for the
extension of VLES turbulence modelling with new turbulent combustion models
together with powerful parallel processing. Another close collaboration was
established with AEA Technology in Munich (Otterfing) regarding the general
purpose software CFX with special emphasis on LES models in combination with
advanced eddy dissipation combustion models and massively parallel processing,
which have recently also been used for engineering studies. Finally, we worked
together with CDL Canada to improve the explosion software for industrial safety
studies. With these productive collaborations it was possible to obtain access to
findings from related research work and to involve recent developments in our
HGF-H2 project work in Part 2.

Cooperation: The HGF-H2 project was performed in cooperation between

FZ Jülich and FZ Karlsruhe, the latter being the coordinator. We received the
deflagration code COM and the detonation code DET from the coordinator for
some test cases performed in Part 2. For this purpose, the detonation code
DET2D was rewritten in Fortran 90 for parallel vector processing (PVP) on the
T90 and using High-Performance Fortran (HPF) for massively parallel processing
(MPP) on the T3E, obtaining quite good performance and scalability for large
numbers of grid sizes and processor nodes. The detonation code DET-2D was
adapted in parameter studies for DDT conditions of mild and strong ignition by
shock wave focusing effects in the RUT test geometry. The deflagration code
COM-3D was used for performance tests on the T3E with varying results and
applied for the numerical simulation of flame acceleration tests with repeated
obstacles performed in the FZK tube. Experimental and numerical results showed

28
 CFD with HPC

tentative agreement, but we used the COM code more or less as a black box, due
to the curtailed cooperation time and lack of data needed for further studies.

Findings: We developed a modern field code cluster (MFCC) with high-

performance supercomputing (HPSC) using the CRAY complex at FZJ, which is
a suitable constellation for improved hydrogen safety analysis of technical
systems related to distribution and combustion effects including explosion aspects
of the transition from deflagration to detonation (DDT). The modern field code
cluster consists of special modules for specific tasks combining applied CFD
methods for practical applications as design or analysis tools together with future
CFD methodology for the next generation of codes as research tools.
Therefore, we also provided higher order turbulence and combustion
models, including large eddy simulations (LES). It is thus possible to obtain a
high numerical resolution of non/reacting flows in complex geometries, depending
on the required details of the flow field. The prototype versions developed were
verified (code-to-code) and validated (code-to-experiment) in benchmark and
experimental tests with fine tuning of the model parameters, obtaining
considerable agreement between experimental and numerical results. Coarse grid
simulations can be performed on fast workstations with single-processing,
whereas fine grid simulations preferably need multi-processing. Moderate parallel
processing is well established, but we also realized a remarkable performance and
scalability of our reactive Navier-Stokes/Euler codes for massively parallel
processing (MPP) on the CRAY-T3E in various performance and optimization
tests.
Comprehensive application test cases for hydrogen distribution and
combustion showed proper iteration and convergence behaviour with sufficient
reduction of the residual to the defined values. Parameter and sensitivity studies
outlined the uncertainties associated with grid-independent solutions for turbulent

29
 CFD with HPC

mixing and turbulent combustion. We found that the eddy dissipation combustion
model generally resulted in appropriate results, but the model constants must be
fitted to the special combustion mode, for example, related to flame acceleration
or hot jet ignition. We assumed that for the flamelet combustion models fewer
empirical contants would be required using standarized burning laws, [Ref. 18,
19].
However, predictions with scaling of turbulent combustion and DDT
processes are still a very complex task and remain an open question for hydrogen
safety applications in real geometries. Concerning DDT conditions, our modern
field code cluster offers the ability to analyse DDT conditions in complex
geometries with regard to critical flame speeds, critical shock Mach number,
critical self-ignition temperatures, onset of detonation formation and propagation
(without the cell structure). More progress has been made in simulating DDT
mode A ahead of the flame front with shock wave focusing and self-ignition in
the unburnt gas compared with DDT mode B in the flame brush with turbulent
mixing and re-ignition triggering DDT processes, which would need further
investigations and huge computer resources. In any case, the numerical simulation
of DDT conditions allows a more objective evaluation basis than in the past in
contrast to the use of simple DDT criteria in a more subjective way (e.g. only
based on detonation cell size classes and geometrical ranking). Such simple DDT
criteria must be carefully used in realistic safety studies, where it is not quite clear
whether or not simple criteria are a conservative approach and where the real
mechanisms are not taken into account. DDT results have shown that very high
reflected detonation pressure peaks can be produced in explosive H2-air mixtures
in the order of about 100 bar and more, which need suitable hydrogen control
systems to limit the resulting combustion loads inside and outside safety
enclosures, Fig. 10. The hydrogen control options are related to mitigation,
prevention, and protection, as discussed below with some pros and cons.

30
 CFD with HPC

Inherent H2 Safety Features for Explosion Protection

Recombiners
Lean Mixtures
4 % H2

Igniters Deflagration: CFX Code

(D2UNS)

9 % H2
Slow Flame

Explosion, DDT:
Hazard AIXCO, SHOCKIN
Regime
Fast Flame

18 % H2
Inertization
Steam Detonation
N2, CO2, He Rapid Flame
DET, IFSAS
Mixing Media
Micro Sprays
58 % H2
Limiting
Conditions
Loads: IFSAS Code
(DYNA)

70 % H2

Hot Jet Injection

H2-Air-Steam

75 % H2
Rich Mixtures

Fig. 10: Hydrogen safety features for fire and explosion protection in technical
systems, consisting of safety measures, including recombiners, igniters, or
inertization to control slow, fast or rapid flames, especially deflagration-to-
detonation transition (DDT) for anticipated events in severe accident conditions,
including predictive CFD methodology for safety assessment

31
 CFD with HPC

Mitigation: The hydrogen safety problem in LWR reactors is associated

with the production of significant quantities of hydrogen-air-steam mixtures
(besides the reactants CO, CO2, etc.) in the case of an extreme core meltdown
accident in the reactor containment due to metal-water and core-concrete
reactions (i.e. in- and ex-vessel reactions). Depending on the specific reactor
design and the anticipated accident scenario, it is to be expected that slow, fast,
or rapid hydrogen flames could develop, jeopardizing the safety equipment and
the safety enclosure. Severe conditions for the reactor containment would occur
during an unlikely transition from deflagration to detonation, initiated by hot jet
ignition, multiple ignition, fan- induced transition or flame acceleration with
shock-wave amplification and detonation formation with end-wall reflection.
To analyse and control hydrogen combustion events, several safety methods
and various safety devices have been studied worldwide as part of the defence-in-
depth concept to reduce the hydrogen risk under severe accident conditions. For
instance, recombiners and/or igniters have been proposed as part of the so-called
dual concept to reduce the amount of free hydrogen or to ignite the flamable
hydrogen at a low concentration level. However, recombiners have a limited
capacity, may be poisoned or could be superheated and igniters could trigger an
explosion under unusual conditions. As a consequense, it is questionable whether
the two safety devices would always be safety-orientated (fail-safe) in realistic
accident sequences. Safety-orientation is a basic principle in nuclear safety
technology! Anyway, recombiners and igniters have special failure rates, which
should be evaluated by fault tree and event tree analysis based on probabilistic
safety analysis before a scientific-technical proven statement could be made about
the efficiency of such devices in reducing the hydrogen risk in comparison with
other core meltdown risks.
In contrast to the recombiner/igniter devices, the dilution/inertization
measures of sensitive hydrogen mixtures act as fire- and explosion-protection

32
 CFD with HPC

systems which are strongly safety-orientated and should be considered for further
investigation and optimization. The diluation/inertization system can be designed
as an active and/or passive safety system, as already used in conventional safety
technology. The amount of inertization defines the quenching of rapid, fast, or
slow hydrogen flames. Furthermore, a cold injection of post-inertization lowers
the containment pressure as part of the accident management strategy (including
water sprays or filtered venting). Finally, an insensitive containment atmosphere
can be handled more easily and safely than a sensitive one. For
dilution/inertization measures, experimental data are available, e.g. for CO2, N2,
He, etc., or modern mixture media for fire protection and explosion prevention.
There exist several safety reasons for partial- or post-inertization measures in
PWR types, e.g. with CO2, whereas N2 pre-inertization is already used in BWR
types during operation (but not in the startup and shutdown period). Last but not
least, dilution/inertization measures make the hydrogen safety problem less
complex and it is therefore more or less reduced to the question of hydrogen
distribution and mixing quality to exclude hydrogen combustion loads in
containments. More recently, modern CFD methods with supercomputing have
been established to predict hydrogen distribution and inertization mixing rates
with a high numerical resolution in complex geometries. Therefore, we improved
our modern field code cluster (MFCC) with high-performance supercomputing
(HPSC) with special emphasis on using classical turbulence modelling for the
mean flow quantities of standard applications as well as introducing large eddy
simulations (LES) for more detailed flow quantities of engineering flows. LES
developments are now powerful numerical tools for fluid flow predictions, which
will become more effective with increasing supercomputer resources towards the
teraflops class in the near future so that more precise numerical simulations will
be possible.

33
 CFD with HPC

Prevention: To control the hydrogen safety problem in light-water reactors,

prevention measures should have the highest priority! Therefore, the reactor
should be designed so that the volume to power ratio is optimized in such a way
that the mean hydrogen concentration in the containment dome is less than about
10 % H2 vol., i.e. below the upper flammability limit, so that DDT events with
self-ignition are normally not to be expected. In this case, global detonations
would be very unlikely as a result of experimental observation with direct ignition
of detonations (e.g. SNL tests). However, non-uniform mixing could cause local
explosions. Therefore, the containment should be gas-tight and capable of
withstanding combustion loads resulting from the complete combustion pressure
(constant volume pressure) for the wet containment atmosphere in the order of the
static design pressure of about 10 bar. As a result, such a passive hydrogen safety
concept would limit maximal possible combustion loads under severe accident
conditions in a credible and convincing manner for a far-reaching consensus from
a generic safety point of view.

Protection: For fire protection and explosion prevention, besides N2/CO2

inertization several new mixing media have been developed more recently and are
already being used in industrial safety, public buildings, and high-tech equipment
to mostly physically suppress fires or explosions. Inertization is a neutralization
of oxygen by partial or total inerting. Modern examples are FM-200,
ARGONITE, INERGEN, MICROSPRAYS, etc. The effectiveness depends on
the special plant design and the specific requirements. For example, FM-200 is a
very effective mixing medium with fast reaction for fire extinction in several
application classes, which would be safe for personal and has been
environmentally proven as well as tested by regulatory authorities. Modern fire
protection and explosion prevention systems could be designed with inherent
safety features as pre-, partial- or post-inertization in technical systems, using fast

34
 CFD with HPC

active or passive injection, which should be also considered for nuclear safety
and hydrogen technology, especially for the combination of nuclear and chemical
power plants (e.g. HTR type with H2 production). Finally, microwater spray
systems can effectively suppress fires and explosions depending on the droplet
size, as experimental observations have shown. Last but not least, water is
relatively cheap and normally available in sufficient quantities. Presently, we
studying recent fire and explosion control systems for more inherent safety (i.e. a
new approach in contrast to risk reduction as acceptably low) in further project
activities.

Conclusion: H2 control with recombiners and igniters (i.e. dual concept)

has special probabilistic failure rates and specific deterministic weak points
initiating unlikely fires or explosions. Self-ignition sources must be avoided as far
as possible, according to the basic explosion protection rules. The major problem
is that the dual system is not absolutely safety-orientated, which is an underlying
principle in nuclear safety. Therefore, it is not quite clear whether the H2 risk is
really reduced by the dual concept. In addition, the dual concept with catalytic
recombiners is always a slow-acting system (in about 24 hours) and it is not very
smart to allow the distribution of a sensitive mixture and to wait for deliberate
ignition thus producing a fire or even an explosion with uncertain consequences
which are more or less unpredictable. Therefore, the dual concept is NOT an
ideal solution to be recommended in nuclear reactor containments for severe
accident conditions without critical review and additional safety measures. Small-
scale laboratory tests and optimistic numerical simulations with simple lumped
parameter codes or simplified CFD simulations on coarse grid levels are not very
representative of the real accident scenario with regard to the completeness of the
accident sequences and scaling uncertainties. As a consequense, appropriate
hydrogen control systems with active and/or passive safety devices should be

35
 CFD with HPC

further established, preferably in combination WITH dilution/inertization

measures using modern mixing media for fire protection and explosion
prevention, to handle the hydrogen safety problem for technical systems in a more
suitable, reliable and inherently safe manner.

Recommendations: Based on our R&D work performed in the scope of

Part 2 of this HGF project for hydrogen safety studies of technical systems, the
following issues should be investigated in more detail with regard to subsequent
project activities:

• Probabilistic safety analysis: We recommend performing a probabilistic

safety analysis based on fault tree and event tree techniques to evaluate the failure
probability and weak points of recombiners and/or igniters under severe reactor
accident conditions in comparison to the estimation of the reliability of modern
dilution or inerting systems providing a scientific-technically proven basis with
quantitative and qualitative assessments of the efficiency of hydrogen control
systems. Probabilistic and deterministic safety studies are therefore proposed
using objective methodologies as far as possible, including worst case
considerations. The results should be used to compare, optimize and harmonize
hydrogen control systems. which are under consideration for various applications
to minimize the probability of a hydrogen risk.

• Fluid dynamic safety analysis: Concerning deterministic safety analysis, we

recommend performing a detailed uncertainty analysis for the prediction of
hydrogen distribution and combustion modes under severe reactor accident
conditions based on systematic parameter and sensitivity studies, including
scaling effects (e.g. turbulent combustion) regarding the completeness of the
accident scenario. To estimate the safety margin of enclosures for combustion

36
 CFD with HPC

loads with the resulting uncertainty band, it is proposed that best-estimate as well
as conservative assumptions should be used. For this reason, a huge number of
large-scale simulations will be necessary, taking into account coarse to fine grid
techniques and modern CFD flow solver algorithms, which should be preferably
executed with high-performance computing and parallel processing to obtain a
sufficiently high numerical resolution. Advanced CFD methods with
supercomputing are already in use for various applications worldwide. Also pre-,
partial- and post-inertization systems with modern mixing media should be
studied in research and practice for fire protection and explosion prevention with
relevance to industrial safety to reduce the accident consequences inside the plant
with a hydrogen risk.

• Structure mechanical safety analysis: Regarding the safety margin, we

recommend that structure mechanical safety analysis should be performed for the
resulting combustion loads (temperature and pressure-impulse history) with
uncertainties under severe reactor accident conditions of safety equipment and
safety enclosures, using finite element (FE) codes. It is suggested that appropriate
CFD codes should be coupled with modern FE codes, which should both be
executed in the massively parallel mode using supercomputing. Nowadays, the
software and hardware infrastructure has been made available for improved
safety studies of nuclear and conventional systems to limit hazard consequences
outside plants with a hydrogen risk.

Comments: The working period of Part2 of the HGF project on hydrogen

safety was from 1998 to 2001. During this time, two projects were performed,
namely the EU-H2DDT project together with the HGF-H2 project, which were
executed by the same CFD working group. The EU-H2DDT project was
completed with a final report in 2000. For the HGF-H2 project, a manpower of

37
 CFD with HPC

2.5 PJ/a was required. During the following working period of the HGF-H2
project, the ISR1 was reorganized at FZ Jülich, which constrained the project
work of Part 2. The reduced working group (1-2 PJ/a) was supplemented by a
guest scientist (8 ma mo) and a post-doc (1.5 PJ) so that the project team was
now, in principle, capable of undertaking the essential working tasks with some
modifications and restrictions. The project work of Part 2 was involved in another
FZJ project for CFD simulations with CRAY supercomputing so that we received
the computer resources for the T90 (50 CPU-h/mo) and for the T3E (2500 CPU-
h/mo), which were sufficient for test cases, but limited for large-scale simulations.
Hence, we performed large-scale simulations in the order of 1 to 10 million grid
points with maximal processor nodes and an effective performance in the MPP
mode.
For the scientific-technical results obtained in Part 2, a special application
plan for further R&D work in research and practice was proposed in the field of
fire protection and explosion prevention for technical systems, e.g. for hydrogen
technology, to continue and apply the findings in a specific project group for
applied computational fluid dynamics (ACFD) with high-performance
supercomputing clusters (HPSC) in FZ Jülich (s. Tab. 2, 3). In this context, the
modern field code cluster (MFCC) developed for supercomputing, networking
and clustering in Europe forms the basis of two recently accepted EC proposals,
i.e. THERMO-NET (nuclear safety) and EXPRO (industrial safety), s. Tab. 4.
The overall project aims are to proceed with the successful project work using
scientific-technical high-performance supercomputing for innovative numerical
simulations of non/reacting flows in complex geometries with modern CFD flow
solvers for the next generation of codes, ranging from the GGG computing level
towards the teraflops computer class. Main results of the EC/HGF-CFD/DDT
project work have been published in international journals and conference papers,
e.g. Ref. [20 - 27].

38
 CFD with HPC

Tab. 2: Recent R&D Work of the ACFD Project Group

Scientific-Technical Flow and Combustion Research

• Higher Order Turbulence and Combustion Models (LES-EDC)
• Fine Structure Models with Massively Parallel Processing (MPP)
• Basic Codes: CFX-LES and AIXCO-VLES with AEA and RWTH
• Research Codes: D2UNS-LES and SHOCKIN-CHEMKIN with INCAS

Basic Scientific Research for Fire and Explosion Protection

• Multi-Fluid Models with Multi-Phase Flows and Particle Transport
• Complex Geometries in Near- and Far-Reaching Fields with Dust Explosion
• Basic Codes: CFX-5 and IFSAS-3D with AEA and CDL

Complex Load Response in Safety Enclosure and Environment

• Fluid-Structure Interaction of Components (IFSAS Code)
• Explosion Loads Inside and Outside Safety Concepts (DYNA Code)

Industrial Safety of Innovative Energy Systems

• Plant Safety: Deflagration-to-Detonation (DDT) Inside and Outside
Safety Buildings for Conventional und Nuclear Energy Systems
• Environmental Safety: Fire/Explosion Protection for Plant Designs (H2)

Scientific Relevance and Practical Competence Centre of Excellence

• Supercomputing with Parallel Processing (CRAY Complex in FZJ)
• Research Clustering (ERCOFTAC Center, CFD Society Canada)
• Application Competence (EC Research, Siemens, Opel, BMW)
• Funding Resources: EC Projects and Proposals (THERMO-NET, EXPRO)

39
 CFD with HPC

Tab. 3: Special ACFD Project Group with Structure

• Applied Computer Fluid Dynamics (ACFD)

in Research and Practice with Supercomputing
Scientific-Technical Flow and Combustion Research
• Computer Fluid Dynamics (CFD)
Non/Reactive Flow Simulations
Intern. Cluster Projects: EC/HGF/DFG/BMBF
Basic Scientific Research for Fire and Explosion Protection
• Fluid Structure Dynamics (FSD)
Interaction and Component Behaviour
Detailed Deformation Analysis and Load Response
Complex Load Calculations in Safety Enclosures
• Application Fields (AF)
Advanced Safety Analysis of the Hydrogen
Behaviour in Technical Plant Systems and the Environment
Industrial Safety of Innovative Energy Systems
• Cooperation (EC)
Universities, Research Centres, Industry
RWTH Aachen, FZ Karlsruhe, AEA Technology, CDL Canada
CEA Grenoble, UPM Madrid, NNC Knutsford, INCAS Bucharest,
Scientific Relevance and Practical Competence
• Collaboration Plan (CP)
Education, Training and Competence
Know-how Transfer and Spin-off/Synergy Effects
Federal Prevention Measures for Accident and Climate Protection

• Organization Plan (OP)

FZJ-NIC Competence Group, IWV Project H2 Fuel Cells
Interdisciplinary CFD Collaboration with Supercomputing
Internal/External Coordination of Computer Fluid Dynamics

40
 CFD with HPC

Tab.4: EC/HGF-CFD/DDT Related Project Activities

Project Title Content Partner

• EC Project: Next CFD Codes Generation and Methodology

H2 DDT Pilot Project FZJ, et al.

Status: Finished 1995 Funding: DM 0.150 mio.
H2 DDT Main Project FUB, FZJ, et al.
Status: Finished 1999 Funding: DM 0.250 mio.
EXPRO CFD-LES UC3, FZJ, et al.
Status: Started 2001 Required: DM 0.400 mio.
Thermo-Net Networking CEA, FZJ, et al.
Status: Started 2001 Required: DM 0.250 mio.
________________________________________________________
EC Brussels

• Project: HPC Parallelization Strategies and Performance

CFD-DDT Supercomputing VSR, ISR, ZAM

Status: Since 1995 T90/T3E Account
________________________________________________________
FZ Jülich

• HGF Project: CFD Software Improvement and Application

H2 Safety Optimization FZK, FZJ

Status: Finished 2001 Required: DM 1.5 mio.
________________________________________________________
HGF Bonn

41
 CFD with HPC

Conclusion and Outlook

In general, fire and explosion protection is an important safety issue in most

areas of the engineering. For instance, the prediction of hydrogen explosion loads
in safety enclosures under severe accident conditions is relevant for nuclear
reactor containments and hydrogen demonstration plants with fuel cells or
hydrogen-powered vehicles. Contextually, special verification tests of reactive
field codes and validation tests of calculations have been performed for specific
reacting flows in complex geometries. The turbulent combustion modes
considered are related to deflagration, transition (DDT), and detonation in
explosive hydrogen-air mixtures. Hereby, experimental and numerical results
were compared on integral- and laboratory scales, obtaining a high temporal and
spatial resolution of complex flows in the numerical simulations. Therefore, a
modern field code cluster (MFCC) was established with new versions of the
reactive Navier-Stokes/Euler flow solver codes: D3UNS, CFX, AIXCO, for fast
flames (deflagrations) and DET, IFSAS, SHOCKIN for rapid flames
(detonations), including vector and parallel processing capabilities. For
benchmark calculations, most of these codes have been ported successfully to the
heterogeneous CRAY-J90/T90/T3E supercomputer complex at FZ Jülich,
running in the sequential, moderate parallel or massively parallel mode with
reasonable performances and good speedup factors.
As a result, the reduction of the computing time per allocated processor
allows appropriate mesh refinement with unstructured or adaptive grids and
robust algebraic multi-grid solvers together with higher order turbulence- and
combustion models of multi-fluid flows and chemical source terms.
Consequently, we have provided RANS/VLES/LES/ flow solver algorithms in
combination with the flamelet concept (FC), the eddy dissipation model (EDM)

42
 CFD with HPC

or probability density functions (PDF). In the scope of joint research project

activities funded by the HGF in Bonn and the EC in Brussels, academic and
commercial software systems (e.g. AIXCO and CFX-5) have been developed for
massively parallel processing (MPP) on the CRAY-T3E with maximum number
of processor nodes (600 MHz) for high-performance supercomputing (HPSC),
based on domain decomposition with message passing tools, using MPI and
PVM or HPF routines. Hence, we use multi-block un/structured pre-processing
(PATRAN/BUILD) as well as online post-processing (AVS/FIELDVIEW) for
meta-supercomputing clusters with huge computer resources, solving the in-
put/output bottleneck of large computing domains within practicable computing
times.
In summary, the applied computational fluid dynamics (ACFD) with
supercomputing makes it possible to explore the scientific-technical aspects of
fire prevention and explosion protection in more detail for realistic safety
enclosures of nuclear and conventional energy systems, especially for the
hydrogen safety technology as a future energy option. Last but not least, the
interdisciplinary collaboration of centers of excellence via high-performance
computer networking (HPCN) contributes to enhance innovative trends of
turbulent combustion simulations in research and practice for engineering flows in
complex geometries.
In this context, we have established a modern field code cluster with hybrid
grids, higher order turbulence and combustion models together with high-
performance computations using supercomputer capabilities at FZJ for detailed
hydrogen safety studies. A major effort and significant progress was made in
introducing large eddy simulations (LES) relevant to industrial safety with the
support of further EC-funded project activities.

43
 CFD with HPC

References

[1] Technical Aspects of Hydrogen Control and Combustion in Severe Light-

Water Reactor Accidents, Ed. by N.C. Rasmussen, et al., National Academy
Press, Washington, 1987.
[2] Hydrogen in Water-Cooled Nuclear Power Reactors, Ed. By EC Brussels and
IAEA Vienna, EUR 14037 EN, 1991.
[3] Hydrogen Behaviour and Mitigation in Water-Cooled Nuclear Power
Reactors, Ed. By E.D. Loggia, EC Brussels, EUR 14039 EN, 1992.
[4] Computational Fluid Dynamics on Parallel Systems, Ed. by S. Wagner, Notes
on Numerical Fluid Mechanics, Volume 50, Vieweg Verlag, 1993.
[5] Large Eddy Simulations of Complex Engineering and Geophysical Flows, Ed.
by S.A. Orszag, et al., Cambridge University Press, 1993.
[6] Turbulent Reacting Flows, Ed. by F.A. Williams, et al., Academic Press,
London, 1994.
[7] Solarwasserstoff – Energieträger der Zukunft, Ed. by P. Hennicke, Wuppertal
Texte, Birkhäuser Verlag, Berlin, 1995.
[8] Compilers for Parallel Computers, Ed. by M. Gerndt, Vol. 21,
Forschungszentrum Jülich, 1996.
[9] Wissenschaftliche Grundlagen des Brand- und Explosionsschutzes, Ed. by S.
Bussenius, Hohlhammer Verlag, Stuttgart, 1996.
[10] EU Research on Severe Accidents, Ed. by G. Van Goethem, et al.,
Symposium FISA 95-97-99-01, EUR 16896 EN, 1996.
[11] Combustion, Ed. by I. Glassman, Academic Press, London, 1996.
[12] Fluid Dynamics and Transport of Droplets and Sprays, Ed. by W.A.
Sirignano, Cambridge University Press, 1999.

44
 CFD with HPC

[13] Supercomputing for Nuclear Applications, Ed. by H. Kaburaki, Intern.

Conference SNA-2000, Tokyo, Sept. 2000.
[14] Process Safety and Industrial Explosion Protection, Intern. ESMG
Symposium 2001, Nuremberg, March 2001.
[15] Numerische Berechnung turbulenter Strömungen in Forschung und Praxis,
Ed. by W. Rodi, University of Karlsruhe, Hochschulkurs 1998.
[16] CFD-2001, Annual Conference of the CFD Society Canada, Waterloo, June
2001.
[17] Computational Physics, Intern. Conference, Aachen, Sept. 2001.
[18] N. Peters: Turbulent Combustion, Edition, RWTH-ITM, Aachen, Oct. 1999.
[19] M. Herrmann: Numerical Simulation of Premixed Turbulent Combustion
Based on a Level Set Flamelet Model, PhD Thesis, RWTH Aachen, 2001.
[20] R. Klein, W. Rehm (Eds.): Models and Criteria for Prediction of
Deflagration-to-Detonation Transition in Hydrogen-Air-Steam Systems under
Severe Accident Conditions, Vol. 9, Forschungszentrum Jülich, 2000.
[21] W. Rehm, M. Gerndt, W. Jahn, R. Vogelsang, B. Binninger, M. Herrmann,
H. Olivier, M. Weber: Reactive Flow Simulations in Complex Geometries with
High-Performance Supercomputing, Proc. Intern Conf. on Supercomputing for
Nuclear Application, SNA-2000, Tokyo, Sept. 2000.
[22] W. Rehm: Angewandte Fluiddynamik mit High-Performance
Supercomputing, ATW Vol. 45 (2000), No. 10, Oct. 2000, Inforum Bonn.
[23] J. Eyink, W. Rehm, et al.: Research on Hydrogen Risk Mitigation Resulting
from Hypothetical Severe Accident, ATW Vol. 45 (2000), No. 12, Dec. 2000,
Inforum Bonn.
[24] W. Rehm, R. Klein: Evaluation of Deflagration-to-Detonation Transition in
Hydrogen-Air Mixtures: DDT-Related Experimental and Numerical Studies,
ATW Vol. 46 (2001), No. 1, Jan. 2000, Inforum Bonn.

45
 CFD with HPC

[25] W. Rehm, C. Nae, B. Wang, W. Jahn: Applied Computational Fluid

Dynamics (ACFD) Using High-Performance Supercomputing Clusters (HPSC),
Proc. Intern. Conf. on Nuclear Reactor Technology, KTG-Forum, Dresden, May
2001.
[26] W. Rehm, C. Nae, W. Jahn, , R. Vogelsang, B. Wang: CFD Simulations of
Turbulent Reactive Flows with Supercomputing for Hydrogen Safety, Proc.
Intern. Conf. on Computational Physics, CCP-2001, Aachen, Sept. 2001.
[27] W. Rehm, et al.: Advanced CFD Methods for Hydrogen Safety Technology
with Supercomputing - Software Optimization and Special Application with
Improvement, Validation and Performance, Report FZJ-ISR1-8/2001, Jülich,
August 2001.

46
 CFD with HPC

Acknowledgments

This R&D work was performed within the scope of European projects on
H2-DDT (Model and Criteria Verification), as well as a FZJ-VSR project on
supercomputing (Reactive Flow Simulations) and of a German HGF-H2 project
(CFD Optimization and Application).

We acknowledge the support of the FZJ (VS, ISR and ZAM). The author is
grateful to all project participants and in particular to the EC in Brussels (Dr. G.
Van Goethem), FU Berlin (Prof. R. Klein), FZ Karlsruhe (Dr. W. Breitung), and
RU-Bochum (Prof. H. Unger) for their cooperation, especially to RWTH-
ITM/SWL Aachen (Prof. N. Peters, Dr. B. Binninger, Dr. H. Herrmann, Prof. H.
Olivier) as well as to AEA Technology (Dr. G. Scheuerer, Dr. F. Unger, Dr.
Kuntz, Dr. Forkel) for their services, including CDL in Canada (Dr. P. Thibault).
We would like to thank everybody involved at FZJ (Dr. M. Gerndt, DI W. Jahn,
Dr. C. Nae, Dr. R. Vogelsang, Dr. B. Wang) for their valuable contributions,
including the administration and infrastucture at FZ Jülich. Finally, we also thank
the ESMG group and KIDDE-DEUGRA GmbH for their helpful information and
material.

Last but not least, I would like to thank cordially my wife Leni-Keil-Rehm
for her permant patience and great interest in preparing the project work with the
documentation for publication, especially all other persons involved for many
productive discussions about further R&D developments.

47
 CFD with HPC

ANNEX: Reactive Field Codes with Supercomputing

Applied Computational Fluid Dynamics (ACFD) Using

High-Performance Supercomputing Clusters (HPSC)

W. Rehm, C. Nae, B. Wang, W. Jahn, FZ Jülich, Germany

Objectives: The overall project aim is to develop modern computational fluid dynamic
methods for hydrogen combustion studies using high-performance supercomputing clusters
with field code optimization and special application, including refinement, validation and
performance analysis. The modern field code cluster developed with parallel processing
capability aims to obtain a high numerical resolution of non/reacting flows in complex
geometries. Special application are related to hydrogen safety and explosion loads under severe
accident conditions for nuclear and conventional energy systems, including transition from
deflagration to detonation processes (DDT), s. Tab. 1 and Figs. 1 to 12, as follows.

Summary: We have developed a modern field code cluster (MFCC) with high-performance
supercomputing (HPSC) using the CRAY computer complex (T90, T3E, J90) at FZ Jülich
with parallel vector processing (PVP) and massively parallel processing (MPP). The MFCC
software system consists of Euler / Navier-Stokes flow solvers: D3UNS, CFX, AIXCO,
SHOCKIN, IFSAS and DET, which are suitable for numerically simulating slow, fast, and
rapid hydrogen flames with resulting combustion loads inside and outside of safety enclosures.
The MFCC software system with supercomputing is used as an advanced research and analysis
tool for fire and explosion protection of innovative energy systems with respect to nuclear
reactor safety and conventional industrial safety (e.g. for hydrogen technology as a future
energy option to limit the increasing climate problem). The R&D work for hydrogen safety at
FZ Jülich is performed in the framework of joint projects supported by the EC in Brussels and
the HGF in Bonn, whose cooperations are gratefully acknowledged.

48
 CFD with HPC

(1) Main Features of the Modern Field Code Cluster

• Global formulation for the Navier-Stokes equations in the conservative variables,
including multifluid non/reacting formulation;
• RANS formulation and spatial filtering for LES using macro redefinitions;
• Implementation of various turbulence models with special wall/near wall treatment;
• Complex boundary condition treatment for real industrial applications;
• Dedicated extensions for special flows (jets, mixing and share layers, free boundaries,
flames);
• Structured/unstructured domain discretization based on multiblocking and domain
decomposition using static/dynamic load balancing;
• Robust mixed finite volume/finite element spatial integration techniques;
• Implicit/explicit time integration techniques using convergence acceleration and
adaptive criteria;
• High level of parallelization and increased efficiency on MPP;
• Integrated solver and pre/postprocessing tools for mesh adaptation and refinement
using flowfield parameters in a closed loop algorithm.
Important CFD criteria and tools implemented in the MFCC for the numerical analysis and
solution assessment (e.g. D3UNS code):
• Controlled level of numerical diffusion and/or dissipation of the numerical scheme;
• True estimator for numerical modeling of viscosity using a global indicator and
distribution analysis;
• Error estimation based on static and/or dynamic definitions using flowfield parameters;
• Wall and near wall assessment of the numerical solution based on complex spatial/time
averaged dedicated analysis;
• Special evaluation and validation of the numerical solution on the boundaries based on
the method of characteristics;
• Complex time/space averaged surface analysis and correlation with experimental data;
• Complex time dependent data analysis;
• Complex and detailed postprocessing tools for solution visualization;
• High level of data exchange with other CFD code packages and general CAD tools;

49
 CFD with HPC

(2) Basic Field Code Equations of the Continuum Physics

∂
∂t
ρ+
∂
∂x j
(
ρ ⋅ u~ j = 0)
∂
∂t
( )
ρ ⋅u~i +
∂
∂x j
( )
ρ ⋅u~i ⋅ u~ j = −
∂
∂xi
Π+
∂ 

∂x j 
( µ + µt )⋅  2 ⋅ S~ij − 2 ⋅ S~ii ⋅ δij 
 3 
∂
∂t
( )
ρ ⋅~
e +
∂
∂x j
[( ) ]
ρ ⋅ e~ + Π ⋅ u~ j =
∂
∂xi
~  ~ 2 ~  ∂
u j ⋅ µ ⋅  2 ⋅ Sij − 3 ⋅ Sii ⋅ δij  + ∂x

( k + kt )⋅
∂ 
Θ
∂x j 
   j 

 µ 
∂
∂t
( )~
ρ ⋅Y +
∂
∂x j
( ~
ρ ⋅ u j ⋅Y =
∂
∂x j
)  ⋅
∂ ~
 ScY ∂x j
Y − ηY , j  + ωY (x, t )

 
where :
C p ⋅µ
Π = ρRΘ 1
Π = p − Dii k=
3 Pr
1
e~ = C v Θ + u~i u~i ~ 1 1
Θ=T −
C p ⋅ µt
2 Dii kt =
2 Cv ρ Prt

and :

~ 1  ∂ ~ ∂ ~  M ij = ρ ⋅ u i ⋅ u j − ρ ⋅ u~i ⋅ u~ j = Tij + Dij

S ij = ⋅  ui + uj
2  ∂x j ∂x i 

1 ~
Tij = M ij − ⋅ M kk ⋅ δ ij = −µ t ⋅ Pij
~ ~ 2 ~ 3
Pij = 2 ⋅ S ij − ⋅ δ ij ⋅ S kk 1
3 Dij = − ⋅ γ ⋅ M SGS
2
⋅ p ⋅ δ ij
3
with:



Φ (U I ,U J ,nIJ ) = F (U I ,nIJ ) + F (U J ,n IJ ) − γ solver ⋅ d ROE (U I ,U J ,n
IJ )
ROE

2
{
C
( )
U I → U IJ = U I + (1 − β )⋅ ∇U + β ⋅ ∇U I ⋅ nIJ
1
2


( )}
D

{
C
( )
U J → U JI = U J − (1 − β ) ⋅ ∇U + β ⋅ ∇U J ⋅ nIJ
1
2


( )}
D

and:
ε NUM
rε (β, γ, ∆, C S ) =
ε SGS

ε NUM =
1
V
(
⋅ (u, v, w )⋅ φ ρu , φ ρv , φ ρw )Tupwind = V1 ⋅ (u ⋅ φ ρu + v ⋅ φ ρv + w ⋅ φ ρw )upwind
2 2
  ~  
= µ t ⋅  2 ⋅ S ij − ⋅ S kk ⋅ δ ij  = ρ ⋅ (C S ∆ ) ⋅ S ⋅  2 ⋅ S ij − ⋅ S kk ⋅ δ ij 
2 2
ε SGS
2

 3   3 

50
 CFD with HPC

(3) Reactive Field Code Equations of Fluid Dynamics

The governing continuum equations of turbulent reactive fluid flows are written in the
following two-dimensional conservative form:

U t + Fx + G y + H = Rx + S y + Q

 ρ  ρu   ρv   ρv 
       
ρ u ρ u 2
+ p ρ vu  ρ 

 
 
 
vu
where U =  ρv  , F =  ρuv  , G =  ρv + p  , H =
2 i  ρv 2 ,
      y 
 ρE
  ρuE +
up  ρvE +
vp  ρvE + vp 
 
 ρY 

 ρuY 
 
 ρvY 
  ρvY

 

 0   0   0 
       Y1   ω1 
τ τ      

 
 
0

 Y2 
 ω 2 
xx xy

R= τ xy , S =  τ yy  
, Q = 0 , Y =
       ...  , ω =  ...  ,
 uτ xx + vτ xy + κTx   uτ xy + vτ yy + κTy   0     
     ρω 

 N
Y ω N 
 0   0   

and τ xx = ( λ + 2 µ )ux + λv y , τ xy = τ yx = µ ( u y + v x ), τ yy = (λ + 2 µ )v y + λu x .

Here p, ρ, T, E, u and v denote pressure, density, temperature, total specific energy and
velocity vector components in the Cartesian coordinates, respectively. Yj and ωj represent the
mass fraction and production rate of jth species. λ and µ indicate coefficients of viscosity, and
κ the thermal conductivity. In the above equations i = 0 and i = 1 correspond to a planar and
an axis-symmetric flow. For the application of a detailed kinetic scheme, the production rate (ω
j) of the jth species is given by:

∑ (ν rjm −ν ljm )(ω fm −ω bm )

m= Nr

m=1

where ν rjm and ν ljm are the coefficients of the jth species in the mth reaction on the right- and

left-hand side, respectively. The forward and backward reaction rates of the mth reaction, ωfm
and ωbm, must be defined by the user.

51
 CFD with HPC

The total specific energy is expressed as:

1 2
E =e+ (u + v 2 ) ,
2
where e denotes the specific internal energy, which can be calculated according to the
thermodynamic database.

α 
ω 
In case of utilising the two-step kinetic model, Y =   and ω =  1  . α and β (0 < α and
β  ω 2 
β < 1) represent the remainder of the progress of the induction reaction and that of the
exothermal reaction. The increasing rates of α and β are expressed as:

dα E
ω1 = = − k1 ρ exp( − 1 ),
dt RT
dβ E E +q
ω2 = = − k 2 p 2 ( β 2 exp( − 2 ) − (1 − β ) 2 exp( − 2 ))
dt RT RT
and the total specific energy is determined by
p 1
E= + qβ + ( u 2 + v 2 )
(γ − 1) ρ 2

where γ indicate the specific heat ratio.

The governing equations can be resolved, e.g. by a second-order extension of Godunov’s

approach, including the Piecewise Linear Method. The details of the applied numerical
approaches for resolving the gas dynamic problem are described in detail by /Yu/. For the
calculations of chemical kinetic and thermodynamic terms, the subroutines of a chemical
kinetic package can be applied. A detailed description of the package is given by /Kee/. In this
context, see for additional reading the following references:

/Yu/: Q. Yu: Experimental and Numerical Study of Unsteady Shock and Detonation Waves,
Doctoral thesis, Stoßwellenlabor, RWTH Aachen, 1996.

/Kee/: R.J. Kee, F.M. Rupley, J.A. Miller CHEMKIN-II: a FORTRAN Chemical Kinetics
Package for the Analysis of Chemical Kinetics, SAND89-8009, 1989.

/B. Wang, W. Rehm/: Numerical Studies of Shock Reflection and Ignition, Intern. Colloqium
on Dynamics of Explosions ICDERS 2001, Seattle, USA, Aug. 2001.

52
 CFD with HPC

Tab. 1: CFD/DDT-related computer models used for hydrogen safety studies at FZJ.

• Distribution Mode (Concentration): Non-reactive Na-St. Solver D3UNS/MPI

Unstructured flow solver with classical and LES turbulence modelling
• Deflagration Mode (Flame Acceleration): Reactive Na-St Solver CFX-3D/MPI
Higher Order Turbulence and Combustion Models with Unstructured Grids
• DDT Mode (Auto-Ignition): Reactive Na-St Solver AIXCO-2D/MPI
Flame Tracking/Shock Wave Capturing Schemes and Reduced Reactions
• DDT Mode (Shock-Ignition): Reactive Euler Solver SHOCKIN-2D/AG
Unstructured Flow Solver with Reduced and Detailed Chemistry
• Detonation Mode (Propagation): Reactive Euler Solver DET-2D/HPF
Hydrodynamics with Chemical Source Term and One Step Reaction Kinetics
• Fluid-Structure Analysis (Loads): Reactive Euler Solver IFSAS-3D/AG
Shock/Detonation Ignition Models with Adaptive Grids and Explosion Loads
• Structure Mechanics (Response): Finite Element Code DYNA-3D/PVM for static
and dynamic deformations of safety enclosures (CONDAT)

Modern CFD Methodology in Fluid Mechanics

Deflagration PVP System MPP System Detonation

HPSC- Clusters
CFX-PVM T90 T3E-600 T3E-1200 DET-HPF
Reactive 10 Proc. Reactive UNICORE Interface
512 Proc. 512 Proc.
Navier-Stokes 8 GBytes 64 GBytes 262 GBytes Euler
18 GFLOPS 300 GFLOPS 614 GFLOPS
Solver Solver FZJ-T3E

MFCC: D2UNS, CFX, COM, AIXCO, DET, DYNA (1) State GMD-SP2

DEC-WS J90 ERCO-DDT FUB-T3E

Pre- and Post- HPSC-
SHOCKIN 16 Proc. Semi-Implicit
Processing 8 GBytes Flame-Shock
IFSAS Networking
BUILD, FIEDVIEW 3 GFLOPS Na-St. Solver RUS-T3E
Fluid Structure Dynamics File Server Transition (2) Plan
Using FZJ Projects: UNICORE (Access), KOJAK (Performance), and GIGABIT TESTBED Meta-Computing

Fig. 1: Modern field code cluster with high-performance supercomputing on the CRAY
computer complex at FZJ for the simulation of reactive flows in complex geometries (e.g.
CFX-PVM and AXICO-MPI on the T3E with massively parallel processing).

53
 CFD with HPC

D E T - 2 D /T 3 E /H P F
H ig h - S p e e d F lo w w ith 5 0 0 . 0 0 0 C e lls

250
T
i 200
m 150
e 100
50
s
0
10 20 30 40 60 80 100
N u m b e r o f P ro c e s s o rs

Fig. 2a: Parallel performance of the reactive Euler solver DET-2D/HPF on the CRAY-T3E
for a detonation test case in a model geometry with obstacles using 500.000 cells.

C FX-3D /T 3E/PVM
C hannel Flow 700.000 C ells

800
Time (s)

600
400
200
0
32 64 96
Processor N odes

Fig. 2b: Parallel performance of the reactive Na-St. solver CFX-3D/PVM on the CRAY-T3E
for a complex flow test case in a tube geometry using 700.000 cells.

A IX C O - 2 D / T 3 E /M P I
B e n c h m a r k te s t
T im e ( s )

400
300
200
100
0
8 16 24 48
N um be r o f P ro c e s s o rs

Fig. 2c: Parallel performance of the reactive Na-St. solver AIXCO-2D/MPI on the CRAY-
T3E for a deflagration test case in a channel geometry with obstacles using 50.000 cells.

54
 CFD with HPC

(4) Non-Reacting Flows (RANS/LES Tests)

Fig. 3: Unsteady iso-vortex representation using RANS code and k-eps turbulence model
(Mach = 0.2, Reynolds = 22.000)

´
Fig. 4: Domain partitioning using load balancing for MPP

Fig. 5: Iso-temperature profile for a 2D jet using LES turbulence model (Mach = 0.10,
Reynolds = 5.000)

55
 CFD with HPC

Fig. 6a: Instantaneous LES µt/µl ratio distribution (Mach = 0.15, Reynolds = 22.000)

r c rite ria distribution

1.E+06
beta = 0.05
beta = 0.10
1.E+05

1.E+04

1.E+03

1.E+02

1.E+01

1.E+00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
r

Fig. 6b: Global statistics for the r criteria distribution for compressible LES simulations

56
 CFD with HPC

Fig. 7: Complex analysis and postprocessing for a 3D LES test case with obstacles

Fig. 8: Instantaneous iso-temperature distribution for accident simulations in open air

57
 CFD with HPC

(5) Reacting Flows (CFD/DDT Tests)

Validation of the SHOCKIN code was performed for experimental results with a series of
schlieren pictures which were obtained in a lean H2-air mixture in a square shock tube with a
long gap in the flat end wall. Unsteady high temperature and density gradients are induced by
shock reflections and by the interaction between a shock wave and a rarefaction fan. The
ignition in such flow fields is an ideal example of validating the reaction model for a wide range
of applications.

Upper wall Gap

End wall
Shock waves

20 µs 20 µs
Lower wall

Reaction wave

40 µs 40 µs

Reaction wave

60 µs 60 µs

Explosion wave

80 µs 80 µs

Detonation wave

Fig. 9: Resolved experi-mental schlieren Fig. 10: Numerically simulated schlieren

pictures of shock-induced ignition at a pictures produced with the reactive fluid
gap entrance (mixture: 15%H2 dynamics combustion code SHOCKIN
+85%air; initial pressure: 4 kPa; initial (conditions and time step correspond to
temperature: 293 K; incident shock Fig. 9). The figure shows the self-ignition
Mach no.: 2.87; gap: hight 10 mm). process and the formation of detonation.

58
 CFD with HPC

The SHOCKIN code was applied to a containment geometry for DDT studies using a two-
step reaction model where a high pressure region is located at the lower corners of the dome.

Fig: 11 : Instantaneous temperature and pressure contours inside the dome

(Mach = 2.85; mixture: 15%H2 +85%air; po: 0.1 MPa; To: 293 K)

(6) Hydrogen Distribution (CFD/BMC Tests)

CFX calculation for a helium injection and distribution test in a multi-compartment geometry
(Batelle Modell Containment).
BMC description :
Volume: 600m³
Air: 1 bar, 304K
Helium Injection:
Diameter: 95mm
Speed : 42m/s
Rate: ca. 0.049 kg/s
Computational details :
Code : CFX with k-eps
Blocks : 91
Elements : 68.522
Points : 99.883 Fig. 12a: BMC model geometry

59
 CFD with HPC

Figs. 12b: Light gas distribution during the first 220 seconds after injection.

Figs. 12c: Light gas distribution during the first 512 seconds after injection.

References

/C. Nae/: Efficient LES using a Betta-Gamma Scheme and Wall Laws, International
Conference on Computational Fluid Dynamics ICFD 2001, Oxford, U.K., 2001.
/B. Wang, W. Jahn, W. Rehm/: Application of a CFD Code for Reactive Flows in H2-Air
Mixtures, CFD 2001 Conference - A CFD Odyssey , Waterloo, Canada, 2001.

60
 CFD with HPC

Nomenclature
a Iteration coefficient
b Pre-exponential factor
c Model constant
c0 Speed of sound
f Fuel fraction
k Turbulent kinetic energy
mlim Minimal mixture fraction
n Normal vector
p Pressure
r Reaction coefficient
sl Laminar flame velocity
st Turbulent flame velocity
t Time
u Fluid velocity
u' Velocity intensity
vj Velocity vector
x Space coordinate
AEA AEA Technology GmbH
AC Combustion constant
AT Burning constant
BR Block ratio
CFDS CFD Society Canada
CDL Combustion Dynamics Ltd.
Cµ Flow constant
D Diffusion coefficient
Da Damköhler number
DNS Direct numerical simulation
E Energy
EC European Commission
EDC Eddy dissipation concept
ERCOFTAC European Research on Turbulent Flow and Combustion
61
 CFD with HPC

F Flux vector
FC Flamelet concept
FZJ Forschungszentrum Jülich
FZK Forschungszentrum Karlsruhe
FUB Freie Universität Berlin
G G equation
ISR Institute for Safety Research and Reactor Technology
ITM Institute for Technical Mechanics
Ka Karlovitz number
MS Shock Mach number
LES Large eddy simulation
Q Iteration variable
R Gas constant
Re Reynolds number
RANS Reynolds-averaged numerical simulation
RWTH Rheinisch-Westfälische Technische Hochschule
S Source term
SWL Shock Wave Laboratory
ST Turbulent burning rate
SC Shock heating rate
T Temperature
Tig Ignition temperature
U Velocity vector
UC3 University Carlos 3, Madrid
UPM University Politecnica, Madrid
V Volume
W Progress variable
ZAM Central Institute for Applied Mathematics
ε Dissipation rate µ Viscosity
µT Turbulent viscosity µE/σf Schmidt-Prandtl No,
ρ Density τ Time
Φ Variable Γ Diffusion term
% H2 Volume fraction of hydrogen

62
 CFD with HPC

Curriculum Vitae

Name: Werner Rehm

Date and place of birth: 10 June 1942 in Nürnberg (Nuremberg)
Marital status: Married to Leni Keil-Rehm
Nationality: German

School qualification: Abitur (University matriculation examination)

Weiden (Opf.), June 1964
University education: Studies of physics (Diplom-Physiker)
RWTH Aachen, October 1970
Special field: Solid state physics and reactor technology
Diploma work: Numerical integration of heat equation
with special consideration of temperature-dependent
heat conductivity
Professional career: Research scientist at FZJ Jülich from Jan. 1971
in the institutes: ZFR, ISF und ISR
Position: Group and project leader of EC/FZJ/HGF projects
Special area: Fluid dynamic safety analysis of technical systems:
hydrogen safety, LWR and HTR with process heat
Dissertation: PhD at RWTH Aachen, Dr. rer. nat., Nov. 1979
on hypothetical accident behaviour of HTRs
Working field: Computer fluid dynamics with hydrogen behaviour
in nuclear and conventional systems (H2 safety)
Special interest: Higher order turbulence and combustion models, ith
high-performance supercomputing cluster (HPSC)
Membership of scientific
associations: ERCOFTAT pilot centre Germany West,
CFD Society Canada (CFD-SC), Reviewer (NED),
international programme committee SNA-2003

63