The Nature of Mathematical Modeling

Neil Gershenfeld

CAMBRIDGE
UNIVERSITY PRESS

r
Contents

Preface 1 Introduction 1.1 Selected References

page x xi 1 3

Part One: Analytical Models 2 Ordinary Differential and Difference Equations 2.1 Linear Differential Equations 2.2 Systems of Differential Equations and Normal Modes 2.3 Laplace Transforms 2.4 Perturbation Expansions 2.5 Discrete Time Equations 2.6 z-Transforms 2.7 Selected References 2.8 Problems 3 Partial Differential Equations 3.1 The Origin of Partial Differential Equations 3.2 Linear Partial Differential Equations 3.3 Separation of Variables 3.3.1 Rectangular Coordinates 3.3.2 Cylindrical Coordinates 3.3.3 Spherical Coordinates 3.4 Transform Techniques 3.5 Selected References 3.6 Problems 4 Variational Principles 4.1 Variational Calculus 4.1.1 Euler's Equation 4.1.2 Integrals and Missing Variables 4.1.3 Constraints and Lagrange Multipliers

5 9 9 12 13 17 18 19 21 22 24 24 26 27 28 29 31 32 33 33 34 34 34 36 37

Contents

4.2 Variational Problems 4.2.1 Optics: Fermat's Principle 4.2.2 Analytical Mechanics: Hamilton's Principle 4.2.3 Symmetry: Noether's Theorem 4.3 Rigid Body Motion 4.4 Selected References 4.5 Problems 5 Random Systems 5.1 Random Variables 5.1.1 Joint Distributions 5.1.2 Characteristic Functions 5.2 Stochastic Processes 5.2.1 Distribution Evolution Equations 5.2.2 Stochastic Differential Equations 5.3 Random Number Generators 5.3.1 Linear Congruential 5.3.2 Linear Feedback 5.4 Selected References 5.5 Problems

38 38 38 39 40 43 43 44 44 46 48 50 51 55 56 57 59 60 60

Part Two: Numerical Models 6 Finite Differences: Ordinary Differential Equations 6.1 Numerical Approximations 6.2 Runge-Kutta Methods 6.3 Beyond RungeKutta 6.4 Selected References 6.5 Problems 7 Finite Differences: Partial Differential Equations 7.1 Hyperbolic Equations: Waves 7.2 Parabolic Equations: Diffusion 7.3 Elliptic Equations: Boundary Values 7.4 Selected References 7.5 Problems 8 Finite Elements 8.1 Weighted Residuals 8.2 Rayleigh-Ritz Variational Methods 8.3 Selected References 8.4 Problems 9 Cellular Automata and Lattice Gases 9.1 Lattice Gases and Fluids 9.2 Cellular Automata and Computing

63 67 67 70 72 76 76 78 79 81 84 91 91 93 93 99 100 101 102 103 107

Contents

9.3 Selected References 9.4 Problems

109 110

Part Three: Observational Models 10 Function Fitting

10.1 Model Estimation 10.2 Least Squares 10.3 Linear Least Squares 10.3.1 Singular Value Decomposition 10.4 Nonlinear Least Squares 10.4.1 Levenberg-Marquardt Method 10.5 Estimation, Fisher Information, and the Cramer-Rao Inequality 10.6 Selected References 10.7 Problems 11 T r a n s f o r m s 11.1 Orthogonal Transforms

111
115
116 117 118 119 122 124 . . . . 125 127 127 128 128

11.2 11.3 11.4 11.5 11.6

Fourier Transforms Wavelets Principal Components Selected References Problems

129 131 136 138 138 139 139 139 141 142 145 147 148 150 152 153 155 155 156 157 161 162 164 166

12 Architectures 12.1 Polynomials 12.1.1 Pade Approximants 12.1.2 Splines 12.2 Orthogonal Functions 12.3 Radial Basis Functions 12.4 Overrating 12.5 Curse of Dimensionality 12.6 Neural Networks 12.6.1 Back Propagation 12.7 Regularization 12.8 Selected References 12.9 Problems 13 Optimization and Search 13.1 Multidimensional Search 13.2 Local Minima 13.3 Simulated Annealing 13.4 Genetic Algorithms 13.5 The Blessing of Dimensionality

Contents

13.6 Selected References 13.7 Problems 14 Clustering and Density Estimation 14.1 Histogramming, Sorting, and Trees 14.2 Fitting Densities 14.3 Mixture Density Estimation and Expectation-Maximization 14.4 Cluster-Weighted Modeling 14.5 Selected References 14.6 Problems 15 Filtering and State Estimation 15.1 Matched Filters 15.2 Wiener Filters 15.3 Kalman Filters 15.4 Nonlinearity and Entrainment 15.5 Hidden Markov Models 15.6 Selected References 15.7 Problems 16 Linear and Nonlinear Time Series 16.1 Linear Time Series 16.2 The Breakdown of Linear Systems Theory 16.3 State-Space Reconstruction 16.4 Characterization 16.4.1 Dimensions 16.4.2 Lyapunov Exponents 16.4.3 Entropies 16.5 Forecasting 16.6 Selected References 16.7 Problems

167 168 169 169 172 174 178 185 185 186 186 187 189 195 197 203 203 204 205 207 208 213 214 216 217 220 224 224

Appendix 1 Graphical and Mathematical Software

Al.l Math Packages Al.1.1 Programming Environments Al.l.2 Interactive Environments A1.2 Graphics Tools Al.2.1 Postscript Al.2.2 X Windows Al.2.3 OpenGL Al.2.4 Java A1.3 Problems

225
226 226 228 230 230 234 240 244 249

Appendix 2 Network Programming A2.1 OSI, TCP/IP, and All That

250 250

Contents

A2.2 Socket I / O A2.3 Parallel Programming

251 254

Appendix 3

Benchmarking

257 259
259 259 266 269 271 276 281 289 292 302 305 309 315 319 323 325
330 340

Appendix 4 Problem Solutions

A4.1 Introduction A4.2 Ordinary Differential and Difference Equations A4.3 Partial Differential Equations A4.4 Variational Principles A4.5 Random Systems A4.6 Finite Differences: Ordinary Differential Equations A4.7 Finite Differences: Partial Differential Equations A4.8 Finite Elements A4.9 Cellular Automata and Lattice Gases A4.10 Function Fitting A4.ll Transforms A4.12 Architectures A4.13 Optimization and Search A4.14 Clustering and Density Estimation A4.15 Filtering and State Estimation A4.16 Linear and Nonlinear Time Series Bibliography Index