See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2281412

Modeling Particle Accelerators using C++ and the POOMA Framework

Article · February 1998

Source: CiteSeer

CITATION READS
1 34

6 authors, including:

Julian Clark Cummings

MSC Software Corporation
67 PUBLICATIONS   948 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

The POOMA Project View project

parallel programming View project

All content following this page was uploaded by Julian Clark Cummings on 20 September 2012.

The user has requested enhancement of the downloaded file.

 Modeling Particle Accelerators using C++ and the POOMA Framework1

Graham A. Mark, William F. Humphrey, Julian C. Cummings,

Timothy J. Cleland, Robert D. Ryne, and Salman Habib
Los Alamos National Laboratory, Los Alamos, NM

1 Introduction
This paper concerns the use of C++ and the POOMA Framework [1] to model high-intensity particle accel-
erators. This work is part of the Computational Accelerator Physics Grand Challenge, sponsored by the U.S.
Department of Energy. Another paper in this conference, \The DOE Grand Challenge in Computational
Accelerator Physics," [2] describes the goals and progress to date of the project.
This Grand Challenge project requires implementation of well-known numerical methods in electromag-
netic particle-in-cell simulation on the latest parallel computer architectures, development of alternative
computational approaches, and smooth interaction of multiple physics packages. For these reasons, we have
chosen to use an object-oriented design for our linear accelerator code. By structuring our model in terms of
abstractions (\objects" and \classes") relevant to accelerator physics, we can develop code that is relatively
easy to understand, maintain and extend.

2 C++ Language Features

C++ has many features that make it an attractive language for object-oriented scientic application codes.
C++ classes provide the means to dene abstractions relevant to a particular problem domain. These classes
can contain both data and methods that act on the data. The contents of a class may be either hidden
from or visible to other code modules, a device that allows class internals to be encapsulated and its external
interface to be xed. In addition, C++ classes can be developed in a hierarchy, so that a child class inherits
properties of parent classes. Inheritance can greatly aid in adapting and specializing an existing class for a
new purpose.
Classes can be created to describe the features and behaviors of physical objects such as charged parti-
cles, particle beams, and beamline elements. Similarly, physics entities such as electric and magnetic elds
and computational objects such as grids for spatial discretization can be represented by classes. A specic
accelerator model may be constructed by creating the appropriate objects with appropriate internal state.
This approach leads to a tremendous amount of exibility and code reuse.
In addition to the object model of programming, C++ oers the capabilities of polymorphism and generic
programming. Polymorphism allows an object to specify its behavior or characteristics when a particular
function acts on it. This permits a very exible style of coding in which methods are invoked on heterogeneous
collections of objects; each object determines how the methods are to do their tasks. Polymorphism is achieved
in C++ through virtual member functions, overloaded functions and operators, and the denition of \traits"
classes using C++ templates. Templates underlie generic programming. They allow the C++ programmer
to parameterize a class or function with an unspecied type. Specifying the parametric type creates a
particular kind of object (an \instantiation"); dierent instantiations result from dierent parametric types.
This technique allows the same piece of code to be reused in many dierent settings. The judicious use of
templates and generic programming can produce compact but highly powerful code.
1
Work supported by the U.S. Department of Energy, Division of Mathematical, Information, and Computational Sciences and
Division of High Energy Physics.
3 Parallel Programming and POOMA
These language features can be used to address a problem known as the \Parallel Platform Paradox": the
time it takes to develop a typical physics application code for the latest supercomputer roughly equals the
lifetime of that computer. This problem exists because custom code must be used if one is to exploit the novel
features of the latest supercomputer. This custom code may include such things as message passing or load
balancing algorithms, as well as architecture-specic data structures or numerical optimizations. Learning
about the new system and writing the code takes time, however, and the newest supercomputer rather quickly
becomes obsolete.
Any given problem domain requires certain commonly used data structures and operations. Representing
these structures and operations eciently in a program often requires substantial amounts of optimized
architecture-specic code. It would make sense to collect these data structures and operations in a C++ class
library, where the optimization and custom coding for the target machine would be done just once. Classes in
the library would provide interfaces for the structures and operations and would simultaneously encapsulate
them, keeping custom code out of the application code.
Another area that often requires custom code is parallelism. All modern supercomputers rely on parallel
processing of some form. Machines dier, however, in exactly how they undertake parallelism and how their
parallel architecture is best exploited. C++'s encapsulation can hide an algorithm's implementation, whether
parallel or serial. Using a library of suitably encapsulated algorithms, the application developer can construct
a physics model without worrying about the exact target architecture. The resulting code should be both
portable and ecient.
POOMA, an acronym for \Parallel Object-Oriented Methods and Applications," is a C++ class library
designed to provide all of these services, and thus to resolve the Parallel Platform Paradox. Application
code that relies on POOMA can be compiled and run without any change wherever POOMA itself exists
on a parallel supercomputer, on a workstation cluster, or on desktop system. The problems of porting
code and of ecient exploitation of each computer system become problems for the maintainers of the
POOMA Framework. The person writing application code can concentrate on physics rather than on machine-
specic programming. This division of labor speeds the development of new applications and broadcasts code
optimizations across a rather broad class of physics codes.

4 Object-Oriented Accelerator Model

We began with a High Performance Fortran program written by R. Ryne and S. Habib. The program models
transport of an intense charged particle beam in a magnetic quadrupole channel. The central routines of the
code follow a collection of particles moving through successive elements in the channel. The electromagnetic
eld of each element, and the beam's self-eld, aect the particles' positions and momenta. The program's
major computational job is integrating the particles forward in time through each of the beamline elements
using the charged-particle equations of motion in an electromagnetic eld.
We dened classes that correspond to the main entities in the model: a class BeamlineElements, with
subclasses Drift and Quadrupole to describe specic types of elements; a class called Beamline, consisting of
a collection of BeamlineElements; and a Beam class that consists of a set of charged particles. To tie it all
together, we created an Accelerator class that contains a Beamline and a Beam and describes our complete
physical system.
Some of these classes { Beamline and BeamlineElement, for example, are useful only in an accelerator
code, and we dened them from scratch for this project. Others, like the Beam class, rely on concepts
that are useful in other kinds of physics applications. This is precisely the sort of general physics-based
abstraction that POOMA provides. POOMA has a base class, ParticleBase, from which our Beam class
was derived. ParticleBase provides a minimal description of a particle collection (a position and ID number
for each particle), along with interfaces for a variety of useful operations such as data-parallel computations
and interpolation to and from a grid. The Beam class inherits these features and adds data specic to our
charged-particle representation.
Another POOMA class of this sort is the Field class, which represents a multidimensional array. The Field
class provides several characteristics usually expected of eld quantities in physics models, such as built-in
boundary conditions, existence on a discretized mesh, and the ability to have scalar, vector or tensor elds.
Moreover, the POOMA Field supports array syntax, stencil operations, dierential operators, and reductions.
In the accelerator code, we use the Field class to represent charge density, electrostatic potential, and the
electric eld. POOMA also contains an FFT class that operates on Fields and is used extensively within the
eld solver portion of the code.
POOMA's ParticleBase and Field classes contain parallel data structures, which are automatically dis-
tributed across processors. By using these classes, we avail ourselves of the many data-parallel operations
that are built into POOMA. We can compile and run our code without change on any platform to which
POOMA has been ported; POOMA will utilize that particular hardware and parallel system as eciently as
possible. In addition to this portable parallelism, POOMA applications such as ours can leverage o of the
many built-in features of the physics-based abstractions contained in the POOMA Framework.

5 Performance Issues
Despite all of these benets, the use of C++ in general and of POOMA in particular would make no sense if
the performance of the resulting code were substantially worse than the performance of equivalent custom-
coded Fortran. Until very recently, numerical codes written in C++ did not perform well in comparison to
equivalent Fortran, but the situation is rapidly changing [3]. One reason for the poor performance of C++
has been the absence of good optimizing compilers. The KCC compiler from Kuck and Associates, Inc. (KAI)
has lled that gap well, and other good optimizing compilers that are fully compliant with the ANSI C++
standard are on the horizon.
Another cause of poor performance is inherent in the C++ language. Consider the following code example:
class Matrix =  :::  =;
Matrix A; B; C; D;
=  :::  =
A = B + C + D; (1)
Suppose that class Matrix overloads the operators \+" and \=" to perform elementwise addition and
assignment. The nal line will be evaluated in a series of binary operations. These will involve temporary
Matrix objects that store intermediate results: tmp1 = B + C ; tmp2 = tmp1 + D; A = tmp2. Creation and
destruction of temporary objects can severely degrade performance, especially if each object contains a lot of
data.
This problem has been recognized for some time, and various attempts have been made to solve it. The
best solution to date is \expression templates" [4], a exible and general device that avoids the creation of
temporaries. POOMA relies heavily on expression templates to optimize data-parallel expressions involving
particles and elds. POOMA applications thereby retain the benets of overloaded operators with no loss in
performance.

6 Project Status
Our goal is to produce a \dimension-independent" linear accelerator model capable of simulating beam be-
havior for a variety of beamline elements. We will use classes that are parameterized by dimension using C++
templates. This means that a single code base will support both 2D and 3D models. (Other dimensionalities
are formally possible but have little practical use.) POOMA provides classes templated on dimension, so our
accelerator code can use this feature and derive templated classes from POOMA classes as needed.
 We have a 2D-prototype code implemented in C++ and POOMA. It supports a K-V or Gaussian initial
beam distribution in the x-y plane and integrates the beam particles through a series of drift and quadrupole
elements. The integration is performed using a split-operator approach. The beam's self-consistent elec-
trostatic potential is computed by scattering charge density into a Field, performing an FFT, applying a
Green's function in Fourier space, and inverting the FFT. POOMA provides simple functions for scattering
the particle charge density, computing the gradient of the electrostatic potential, and gathering the resulting
electric eld at the particle positions.
Our results are in agreement with results of Ryne and Habib's 2D HPF code. The POOMA code is
instrumented to send particle and eld data to ACLVIS, a Los Alamos visualization package, during a code
run. This provides real-time data visualization capabilities that enable users quickly to spot problems in
code behavior and to study the eects of various beamline elements. Furthermore, POOMA provides a
simple mechanism for proling application codes with the Tau proling tools [5]. Simple macros in the
accelerator code generate timing data. Tau uses the data to chart the CPU time spent in each instrumented
routine by each processor. We are using these proling tools to analyze the performance of our code, and to
compare it with the performance of the HPF code. Our most recent tests, run on an Origin 2000 symmetric
multiprocessor computer, indicate that the POOMA code is comparable to the HPF code. More studies need
to be done before specic performance data can be provided. Our future work includes such studies and
recasting the current code into a generic templated form.

References
[1] J. V. Reynders, V. W. John, P. J. Hinker, J. C. Cummings, S. R. Atlas, S. Banerjee, W. F. Humphrey, S.
R. Karmesin, K. Keahey, M. Srikant, M. Tholburn, in Parallel Programming Using C++, G. V. Wilson
and P. Lu, eds. (MIT Press, Cambridge, 1996).
[2] R. D. Ryne, S. Habib, K. Ko, Z. Li, W. Mi, C.-K. Ng, J. Qiang, M. Saparov, V. Srinivas, Y. Sun, X.
Zhan, Proceedings ICNSP'98.
[3] T. Veldhuizen, http://monet.uwaterloo.ca/~tveldhui/DrDobbs2/drdobbs2.html
[4] T. Veldhuizen, C++ Report 7:5, 26 (June, 1995). Reprinted in C++ Gems, Stanley B. Lippman, ed.
(Sigs Books, NY, 1996).
[5] Tau. http://www.acl.lanl.gov/tau/

View publication stats