Escolar Documentos
Profissional Documentos
Cultura Documentos
Outline
Introduction Grid Management Linear Solver Pre-processing Visualization Numerical Experiments
Introduction: Motivation
Desktop/workstation Small cases, efficient Large cases, tens of millions of grid cells Large number of wells Multi-core, OpenMP
Our Methods
Cluster OpenMP - multi-core, easy to use, limited scalability MPI - communication - MPI-IO C language
Typical Simulator
Grid Physics
Data
SIM
Visualization
Solver
Key words
IO
Communication
Memory
Development Goals
Common platform for various simulators - Grid, data, linear solver and IO - Pre- and post-processing Hundreds of millions of grid cells (or more) Hundreds of processors (or more)
Grid Management
The most critical module - Type, choice of numerical methods - Partition, workload, communication - Data structure, info and data distribution - Numbering, bandwidth, input
Grid Type
Hexahedron Structured (FD and FV) Unstructured (to be implemented)
Grid Partition
Topological - Connection, dual graph - METIS, ParMETIS Geometry - Location info, coordinate (centroid) - Zoltan
Parallel BLAS
Distributed matrix Distributed vector Matrix and vector management Parallel matrix & vector operations (BLAS 1/2) Global communication modules
Linear Solvers
Solver - GMRES(m) , ORTHOMIN(m) - BICGSTAB, CG , CGS - AMG (From Hypre) Preconditioner - Domain decomposition preconditioner - ILU(k), ILUT and Direct method - AMG - Constrained pressure residual (CPR)
Pre-processing
Define key words and well file Choose model Setup reservoir and well data Setup initial and boundary condition Setup parameters
Pre-processing
Set initial condition, read from file - Saturation - Porosity - Permeability, etc Parallel read, MPI-IO Distributed data among processors
Visualization
Display grid, well and physics properties Output format, structured and unstructured grid Parallel read/write operations Graphics engine, rendering
Visualization: VTK
VTK toolkit, graphics engine VTK formats Parallel write, MPI-IO Paraview displays VTK file
Visualization: Example
num
ana
Visualization: Example
Numerical Experiments
Parallel (U of Calgary), Westgrid 528 standard nodes - 2 6-core Intel Xeon E5649 processors - 24 G memory InfiniBand 4X QDR, 40 Gbit/s
2
44.93
4
21.74 59.82
8
10.63 27.48
16
5.36 13.54
32
2.81 6.90
64
1.59 3.49
16 49.19 1258.55
32 24.18 662.10
64 11.84 338.68
16 80.54 2471.98
32 38.93 1286.08
64 19.82 670.83
Conclusion
A parallel platform for reservoir simulation has been developed. The platform supports finite difference method and finite volume method. The platform has the capability to calculate very large problems.
Future Work
Serious tests, using more processors Optimize Develop reservoir simulators Develop physics-based preconditioners
Hui Liu
hui.j.liu@ucalgary.ca
http://schulich.ucalgary.ca/chemical/JohnChen
Sponsors