Numerical Methods and Analysis

Part I Roots of Equations

The root of an equation is the value of x that makes f(x)=0. Roots are sometimes called the zeros of the equation.

Chapter 1 Bracketing Methods

These methods exploit the fact that a function typically changes sign in the vicinity of a root. They are called bracketing methods because two initial guesses for the root are required.

Bisection Method
Also known as Binary Chopping, this incremental search method locates the interval where the function changes sign by dividing the interval into two sub-intervals. Each of these sub-intervals is searched to locate the sign change. The method uses an initial bracket [a b] that contains the root x. By continuously bisecting the bracket, the midpoint m of the bisection approximates the root x of the function. Shown below is the graph of the function f(x) = x2 3x + 1. From inspection, we find that the roots of the function (i.e., the values of x for which f(x) is 0) lie somewhere around x=0.5 and x=2.5, respectively.

Using Bisection Method to estimate the first root (around x=0.5), the following are the steps: 1. Make an initial guess of an interval [a b] that necessarily contains the first root, say, [0 1] (we cannot use [0 3] since it also covers the second root). 2. Bisect the interval and locate the midpoint, m. 3. Solve for f(a), f(b) and f(m): a 0 m 0.5000 b 1.0000 f(a) 1.0000 f(m) -0.2500 f(b) -1.0000

4. We must choose the interval for the next iteration from between the two subintervals [a m] and [m b]. Notice that the sub-interval [a m] does contain the zerocrossing since the function changes sign from f(a) to f(m), i.e., from positive to negative, passing through zero (if neither or both sub-intervals contain a zerocrossing, then the initial interval does not contain any root or does contain more than one root and we must look for another initial interval). Hence, the root is found in the sub-interval [a m] rather than in [m b] which does not contain the zero-crossing (the function does not change sign from f(m) to f(b)). Therefore our interval for the next iteration is [a m], i.e., the old [a m] becomes the new [a b]: a 0 0 m 0.5000 b 1.0000 0.5000 f(a) 1.0000 f(m) -0.2500 f(b) -1.0000

5. Solve for the new midpoint and complete the table. After 5 iterations we will get: a 0 0 0.2500 0.3750 0.3750 m 0.5000 0.2500 0.3750 0.4375 0.4063 b 1.0000 0.5000 0.5000 0.5000 0.4375 f(a) 1.0000 1.0000 0.3125 0.0156 0.0156 f(m) -0.2500 0.3125 0.0156 -0.1211 -0.0537 f(b) -1.0000 -0.2500 -0.2500 -0.2500 -0.1211

6. Our answer so far, therefore, is x=0.4063. 7. If we continue with more iterations, we will reach m=0.3820, which is the true value of the first root. Therefore, our first root is x=0.3820. To get the second root (around x=2.5), we can use the initial interval [1 3] (we cannot use [0 3] since it also covers the first root). Here are the results:

a 1.0000 2.0000 2.5000 2.5000 2.5000

m 2.0000 2.5000 2.7500 2.6250 2.5625

b 3.0000 3.0000 3.0000 2.7500 2.6250

f(a) -1.0000 -1.0000 -0.2500 -0.2500 -0.2500

f(m) -1.0000 -0.2500 0.3125 0.0156 -0.1211

f(b) 1.0000 1.0000 1.0000 0.3125 0.0156

Our answer so far is x=2.5625. If we continue, we will reach m=2.6180, which is the true value of the second root. Hence, our second root is x=2.6180. Using 4 decimal places, the true value of the first root appeared after 13 iterations, while the second appeared after 14 iterations. For other functions the true values of the roots may appear after 100 iterations or more. There are various criteria to stop the computation after coming up with a fairly acceptable estimate of the root. The criterion will be specifically provided for in the problem. For example, if the problem requires you to stop the computation when: |f(x)|<0.1, then, since the root is estimated by the midpoint, you stop when |f(m)|<0.1. Therefore, our answers are x=0.3750 (3rd iteration) and x=2.6250 (4th iteration), respectively. Drill: Solve for the roots of f(x) = 2x2 - 2x - 3. Assume that the roots are bounded by [0 2]. Stopping criterion: |f(x)|<0.1. Answer: x=1.8125 (5th iteration) and x=-0.8125 (5th iteration)

False Position (Regula Falsi) Method

Also known as Linear Interpolation, this method proceeds the same way as the Bisection Method, except that the interval is not bisected and, therefore, there is no midpoint that approximates the root. The graph given below is the same as above except that only the first root is shown. Our initial interval [a b] is still [0 1]. The point (c,0) lies along the line connecting the points (a,f(a)) and (b,f(b)), intersecting the x-axis. This intersection represents an improved estimate of the root. The name originates from the fact that the replacement of the curve by a straight line gives a false position of the root.

We choose the interval for the next iteration from between the two sub-intervals [a c] and [c b]. As the interval gets smaller, c approximates the root of the function. Using similar triangles, we can solve for c. For the first root, the results for 5 iterations are given below: a 0 0 0 0 0 c 0.5000 0.4000 0.3846 0.3824 0.3820 b 1.0000 0.5000 0.4000 0.3846 0.3824 f(a) 1.0000 1.0000 1.0000 1.0000 1.0000 f(c) -0.2500 -0.0400 -0.0059 -0.0009 -0.0001 f(b) -1.0000 -0.2500 -0.0400 -0.0059 -0.0009

For the second root, the results for 6 iterations are given below: a 1.0000 2.0000 2.5000 2.6000 2.6154 2.6176 c 2.0000 2.5000 2.6000 2.6154 2.6176 2.6180 b 3.0000 3.0000 3.0000 3.0000 3.0000 3.0000 f(a) -1.0000 -1.0000 -0.2500 -0.0400 -0.0059 -0.0009 f(c) -1.0000 -0.2500 -0.0400 -0.0059 -0.0009 -0.0001 f(b) 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000

Using the same stopping criterion given earlier, our answers are x=0.4000 (2nd iteration) and x=2.6000 (3rd iteration), respectively. If we compare this result with the number of iterations required in the Bisection Method to satisfy the stopping criterion (3 and 4 iterations, respectively), we observe that the False Position Method gives a faster convergence in the computation for the roots. Observe that the False Position Method requires 5 and 6 iterations, respectively, to get to the true values of the roots. The Bisection Method requires 13 and 14 iterations, respectively. The difference can be explained by the fact that the Bisection Method approximates the root roughly as the midpoint of the bisection of the bracket while the False Position Method uses a more sophisticated approximation. Drill: Solve for the roots of f(x) = 2x2 - 2x - 3. Assume that the roots are bounded by [0 2]. Stopping criterion: |f(x)|<0.1. Answer: x=1.8214 (3rd th iteration) and x=-0.8122 (4 iteration)

Chapter 2 Open Methods

In bracketing methods, the root is located within an interval. Repeated application of these methods always results in closer estimates of the true value of the root. Such methods are said to converge because they move closer to the truth as the computation progresses. In contrast, open methods are based on formulas that require a single starting estimate of the root or two starting estimates that do not necessarily bracket the root. As such, they sometimes diverge or move away from the true roots. However, when open methods converge, they usually do so much more quickly than the bracketing methods.

Simple One-Point Iteration Method

This method employs a formula by rearranging the function f(x)=0 so that x is on the left-hand side of the equation. This transformation can be accomplished either by algebraic manipulation or by simply adding x to both sides of the original equation. For example, f(x) = x2 3x + 1 can be transformed by transposing -3x to the left and dividing the equation by 3. With an initial estimate of x=0, we can replace this value to the transformed equation to obtain a better estimate of the root. The first 5 results are: x 0 0.3333 0.3704 0.3791 0.3812 f(x) 1.0000 0.1111 0.0261 0.0065 0.0017

Using the stopping criterion given earlier, the root is x=0.3704. Observe how fast the function approaches zero. Using the stopping criterion given earlier, we stop computing at the 3rd iteration. We can also transform the original equation by transposing x2 to the left, multiplying the equation by -1 and getting the square root of both sides. With an initial estimate of x=5, the first 5 results give us an estimate of the second root. x 5.0000 3.7417 3.1977 2.9314 2.7918 f(x) 11.0000 03.7750 01.6320 00.7988 00.4187

Using the same stopping criterion, we do not have an answer given only 5 iterations. Drill: Solve for the roots of f(x) = x2 3x + 1 using one-point iteration method by adding x to both sides of the equation. Use an initial estimate of x=0. Answer: Divergent! x 0 1 0 1 0 f(x) +1 -1 +1 -1 +1

Newton-Raphson Method
Shown below is the graph of the function f(x) = x2 3x + 1. If the initial estimate of the root is x=a, we define a point along the curve (a,f(a)) as a point of tangency and draw a line tangent to the curve at the said point, intersecting the x-axis at x=b. Using the relationship between the curve and the slope of the line tangent to it, we can solve for b which gives a better estimate of the root. Repeating the computation will move b closer each time to the true value of the root.

Using an initial estimate of x=5, the first 5 results are: x 5.0000 3.4286 2.7884 2.6293 2.6181 f(x) 11.0000 02.4694 00.4099 00.0253 00.0001

Using the stopping criterion given earlier, the root is x=2.6293. For the second root, with initial estimate x=0, the first 5 estimates are: x 0 0.3333 0.3810 0.3820 0.3820 f(x) 1.0000 0.1111 0.0023 0.0000 0.0000

Using the same stopping criterion, the root is x=0.3810. Drill: Solve for a root of f(x) = 2x2 - 2x - 3 using an initial estimate of x=0 and the same stopping criterion. Answer: x=-0.8274 (4th iteration)

Secant Method
This method does away with derivatives but requires two initial estimates of the root. However, these estimates do not have to form a bracket that contains a root. Shown below is the graph of the function f(x) = x2 3x + 1. If the initial estimates of the root are x=a and x=b, respectively, we define two points along the curve, (a,f(a)) and(b,f(b)), and draw a line touching the curve at these points, intersecting the x-axis at x=c. Using similar triangles we can solve for c which gives a better estimate of the root. For the next iteration do the following substitutions: a=b and b=c. Repeating the computation will move c closer each time to the true value of the root.

Using initial estimates a=0 and b=1, the first 5 results are: x 0.5000 0.3333 0.3846 0.3820 0.3820 f(x) -0.2500 +0.1111 -0.0059 -0.0001 +0.0000

Using the stopping criterion given earlier, the root is x=0.3846. For the second root, with initial estimates a=4 and b=5, the first 5 estimates are: x 3.1667 2.8710 2.6637 2.6226 2.6181 f(x) 1.5278 0.6296 0.1042 0.0102 0.0002

Using the same stopping criterion, the root is x=2.6226.

Drill: Solve for a root of f(x) = 2x2 - 2x - 3 using initial estimates of x=1 and x=2 and the same stopping criterion. Answer: x=1.8182 (2nd iteration)

Newton Method
This method is a modification of the Newton-Raphson Method and is applicable only to solving the square root of a non-negative real number. The formula for the estimates can be derived from the Newton-Raphson formula. If a is a real number, then we can estimate its square root as xo. If we divide a by such estimated square root, the quotient, i.e., a/xo, is another estimated square root of the number. Getting the average of the two estimates gives us an improved estimate of the square root. Solving for the square root of 3 with an initial estimate of x=1, the first 5 results are as follows: xo 1.0000 2.0000 1.7500 1.7321 1.7321 f(xo) -2.0000 +1.0000 +0.0625 +0.0003 +0.0000

Using the stopping criterion given earlier, the answer is x=1.7500 which is one iteration away from the true value x=1.7321. Drill: Solve for a square root of using initial estimate x=1 and the same stopping criterion. Answer: x=1.7939 (3rd iteration) Drill: Derive the Newton Formula.

Part II Systems of Linear Algebraic Equations

A system of linear algebraic equations can be expressed as a single equation in matrix form: Ax = b where A is the coefficient matrix, x the unknown matrix and b the right-hand matrix. Thus, the system: -12a + b - 8c a - 6b + 4c -2a - b + 10c can be expressed as: = -80 = 13 = 90

where A is

x is

and b is

Chapter 1 Determinants
Cramers Rule
This rule states that the its ith unknown in a system of linear algebraic equations may be expressed as a fraction of two determinants with denominator D and numerator Di, where D is the determinant of the coefficient matrix and Di the determinant of the coefficient matrix with its ith column replaced with the right-hand matrix. Hence, given the system described above: D D1 D2 D3 Therefore: = 758 = 494 = 3,272 = 7,248 a = 0.6517 b = 4.3166 c = 9.5620

Drill: Solve the following system of linear algebraic equations: 2a - 5b + -a + 3b 3a - 4b + Answer: D D1 D2 D3 a b c = = = = 4 8 -4 12 c c 2c = = = 12 -8 16

= 2 = -1 = 3

Chapter 2 Gauss Elimination Method

Naive Gauss Elimination Method
This method starts with appending the right-hand matrix to the coefficient matrix to form an n by n+1 matrix, where n is the number of unknowns. Hence, given the same system, we have the matrix:

The method proceeds with forward elimination by eliminating the elements below the diagonal using the elementary row operations. The following are the elementary row operations: 1. multiplying with or dividing an equation by any constant; 2. adding to or subtracting one equation from another; and 3. interchanging any two equations. These operations preserve the system. The elements along the diagonal are indicated in red color as shown below. Hence, the element to be eliminated in the second row is 1, while the elements in the third row are -2 and -1.

Using the elementary row operations between row1 and row2 and applying the result to row2, we have:

Using the elementary row operations between row1 and row3 and applying the result to row3, we have:

Using the elementary row operations between row2 and row3 and applying the result to row3, we have:

The final step is backward substitution, starting with the last row to solve for the last unknown:

Therefore:

c = 9.5620

b = 4.3166

a = 0.6517

Drill: Solve the same system of linear algebraic equations. Answer: 2.0000 0 0 a = 2 -5.0000 1.0000 0 1.0000 -1.0000 -1.1429 b = -1 12.0000 -4.0000 -3.4286 c = 3

Chapter 3 Gauss-Jordan and the Matrix Inverse Methods

Gauss-Jordan Method
This method starts with appending the right-hand matrix to the coefficient matrix to form an n by n+1 matrix, where n is the number of unknowns. Hence, given the same system, we have the matrix:

It proceeds by converting the original coefficient matrix into an identity matrix using the elementary row operations. We can take the course via Gauss Elimination and using the result above we have:

Then use the elementary row operations to convert it to:

And eventually to:

The last column of the resulting matrix gives us the solution. Therefore, a b c = 0.6517 = 4.3166 = 9.5620

Drill: Solve the same system of linear algebraic equations. Answer: 1.0000 0 0 a = 2 -2.5000 1.0000 0 0.5000 -1.0000 1.0000 b = -1 6.0000 -4.0000 3.0000 c = 3

Matrix Inverse Method

As discussed above, a system of linear algebraic equations can be expressed as a single equation in matrix form: Ax = b To solve for the solution matrix x, we pre-multiply the equation by the inverse of A: A-1 (Ax) (A-1A) x (I) x x = = = = A-1 A-1 A-1 A-1 (b) (b) (b) (b)

Hence, the solution matrix is the product of the inverse coefficient matrix and the right-hand matrix. We can use the Gauss-Jordan Method to solve for the inverse coefficient matrix by appending an identity matrix to the coefficient matrix to form an n by 2n matrix, where n is the number of unknowns. Hence, given the same system, we have the matrix:

We then use the elementary row operations to convert the coefficient matrix into an identity matrix. Hence, we have:

The right half of the resulting matrix gives us the inverse coefficient matrix which we multiply with the right-hand matrix. Thus:

We multiply the first row of the first matrix by the first column of the second matrix: (-0.0739)(-80)+(-0.0026)(13)+(-0.0580)(90) = 0.6517 and so on and so forth. Therefore, a b c = 0.6517 = 4.3166 = 9.5620

Drill: Solve the same system of linear algebraic equations. Answer: A-1 = 0.5000 -0.2500 -1.2500 a = 2 b = -1 c = 3 1.5000 0.2500 -1.7500 0.5000 0.2500 0.2500

Chapter 4 Gauss-Seidel and Jacobi Methods

Gauss-Seidel Method
This method is an extension of the One-Point Iteration Method discussed earlier. Hence, using the system described above, we have the following equations: a = (80 + b 8c)/12 b = (-13 + a + 4c)/6 c = (90 + 2a + b)/10 We may use initial estimates of 0, 0 and 0, respectively. We compute for the next estimate of a using the latest values of b and c, which are 0 and 0, respectively. Hence, we have: a 0 6.6667 b 0 c 0

We compute for the next estimate of b using the latest values of a and c, which are 6.6667 and 0, respectively. Hence, we have: a 0 6.6667 b 0 -1.0556 c 0

We compute for the next estimate of c using the latest values of a and b, which are 6.6667 and -1.0556, respectively.

Hence, we have: a 0 6.6667 b 0 -1.0556 c 0 10.2278 The

We continue to the next row by using the latest values of each of the variables. following are the first 10 results: a 0 6.6667 -0.2398 0.7755 0.6339 0.6542 0.6514 0.6518 0.6517 0.6517 b 0 -1.0556 4.6119 4.2381 4.3249 4.3152 4.3168 4.3166 4.3166 4.3166 c 0 10.2278 9.4132 9.5789 9.5593 9.5624 9.5620 9.5620 9.5620 9.5620

We observe that all the estimates no longer change in the 10th iteration. Therefore, a b c = 0.6517 = 4.3166 = 9.5620

Drill: Solve the same system of linear algebraic equations. Answer: a 0 6.0000 5.5000 3.7083 0.8993 -2.2451 -4.7605 -5.6513 -4.2144 -0.3468 b 0 -0.6667 -1.6111 -2.5880 -3.2797 -3.3845 -2.7206 -1.3173 0.5427 2.3534 c 0 -2.3333 -3.4722 -2.7384 0.0916 4.5987 9.6995 13.8425 15.4070 13.2270

Observe that the solution does not seem to converge after 10 iterations. In fact, it does not converge after 100 iterations. Even if we interchange any two equations, the solution is still divergent.

Jacobi Method
This method is a slight modification of the Gauss-Seidel Method. The only difference lies in the values used in each computation. While the Gauss-Seidel Method uses the latest values of each of the variables, the Jacobi method uses the values in the immediately preceding row, regardless of whether or not they are the latest values. Hence, we compute for the next estimate of a using the values of b and c found in the first row which are 0 and 0, respectively. We have: a 0 6.6667 b 0 c 0

We compute for the next estimate of b using the values of a and c found in the first row which are 0 and 0, respectively. Hence, we have: a 0 6.6667 b 0 -2.1667 c 0

We compute for the next estimate of c using the values of a and b found in the first row which are 0 and 0, respectively. Hence, we have: a 0 6.6667 b 0 -2.1667 c 0 9.0000

We continue to the third row by using the values of each of the variables in the second row. The following are the first 15 results: a 0 6.6667 0.4861 0.3343 0.6605 0.6685 0.6513 0.6508 0.6517 0.6518 0.6517 0.6517 0.6517 0.6517 0.6517 b 0 -2.1667 4.9444 4.6588 4.2835 4.2986 4.3184 4.3176 4.3165 4.3166 4.3166 4.3166 4.3166 4.3166 4.3166 c 0 9.0000 10.1167 9.5917 9.5327 9.5604 9.5636 9.5621 9.5619 9.5620 9.5620 9.5620 9.5620 9.5620 9.5620

We observe that all the estimates no longer change in the 12th iteration. Therefore, a = 0.6517 b = 4.3166 c = 9.5620

Drill: Solve the same system of linear algebraic equations. Answer: a 0 6.0000 -4.6667 14.1667 -19.3333 b 0 -2.6667 2.0000 -6.3333 8.3889 c 0 8.0000 -6.3333 19.0000 -25.9167

Observe that the solution does not seem to converge after 5 iterations. The solution is divergent.

Chapter 5 LU Decomposition
Crout LU Decomposition Method
This method decomposes the coefficient matrix A into the product of a lower triangular matrix L and an upper triangular matrix U. Hence: A = LU where A, L and U are expressed as: A = A11 A21 A31 A12 A22 A32 A13 A23 A33 L = L11 L21 L31 0 L22 L32 0 0 L33 U = 1 0 0 U12 1 0 U13 U23 1

Using our original equation: Ax = b We substitute: (LU) x L (LU)x (L-1 L) Ux (I) Ux Ux U-1 (Ux) (U-1 U)x (I)x x
-1

= = = = = = = = =

b L-1 L-1 L-1 L-1 U-1 U-1 U-1 U-1

b b b b (L-1 (L-1 (L-1 (L-1

b) b) b) b) eqn. 1

We let: d L(d) Ld Ld Ld = = = = = L-1 b L(L-1 b) (LL-1)b (I) b b eqn. 2

Then from eqn. 1 : x (U)x (U)x (U)x Ux = = = = = U-1 d (U) U-1 d (UU-1)d (I) d d eqn. 3

Now, using the same system of linear algebraic equations given earlier, we have: A =

b=

Generally, if we multiply L by U we will get A. If we multiply the first row (r=1) of L by the first column (c=1) of U, we get A11 ( i.e., Arc ) : A11 = L11(1) + 0(0) + 0(0) Hence, L11 = -12 If we continue, we will get the following results: L = -12.0000 1.0000 -2.0000 U = 1 0 0 -0.0833 1 0 0.6667 -0.5634 1 0 -5.9167 -1.1667 0 0 10.6761

Using eqn. 2, Ld = b, we have: L -12.0000 +01.0000 -02.0000 0 -5.9167 -1.1667 0 0 10.6761 d d1 d2 d3 = b -80 +13 +90

If we do the multiplication, we will get the following results: d = +6.6667 -1.0704 +9.5620

Using eqn. 3, Ux = d, we have: U 1 0 0 -0.0833 1 0 0.6667 -0.5634 1 x x1 x2 x3 = d +6.6667 -1.0704 +9.5620

If we do the multiplication, we will get the following results: x = 0.6517 4.3166 9.5620

Therefore, a b c = 0.6517 = 4.3166 = 9.5620

Drill: Solve the same system of linear algebraic equations. Answer: L = 2.0000 -1.0000 3.0000 U = 1.0000 0 0 d = 6 -4 3 -2.5000 1.0000 0 a = 2 b = -1 c = 3 0.5000 -1.0000 1.0000 0 0.5000 3.5000 0 0 4.0000

Matrix Inverse by LU Decomposition Method

The LU Decomposition can also be used to compute the inverse of the coefficient matrix. Using the same system given earlier we have: L = -12.0000 1.0000 -2.0000 U = 1.0000 0 0 -0.0833 1.0000 0 0.6667 -0.5634 1.0000 0 -5.9167 -1.1667 0 0 10.6761

We now solve for the first column of the inverse coefficient matrix: x1 x2 x3 The formula is: eqn. 1 xn = dn xi = di - ( Uij xj )

for i = n-1, n-2,. . ., 1

eqn. 2

where n is the number of unknowns in the system and the summation index j takes values from i+1 to n. The matrix d is solved from: L (d ) = 1 0 0 1 0 0

d1 d2 d3

If we do the multiplication, we will get the results: d1 = -0.0833 d2 = -0.0141 d3 = -0.0172 Applying eqn. 1, we have: x3 = -0.0172

Applying eqn. 2, we have: x2 = -0.0237 x1 = -0.0739

Therefore, the first column of the inverse coefficient matrix is: -0.0739 -0.0237 -0.0172 We can solve for the second column of the inverse coefficient matrix using: L (d ) = 0 1 0

If we do the multiplication, we will get the results: d1 = 0 d2 = -0.1690 d3 = -0.0185 Further, we have the results: x3 = -0.0185 x2 = -0.1794 x1 = -0.0026 Therefore, the second column of the inverse coefficient matrix is: -0.0026 -0.1794 -0.0185 We can solve for the third column of the inverse coefficient matrix using: 0 0 1 Then we will get the results: d1 = 0 d2 = 0 d3 = 0.0937 Further, we have the results: x3 = 0.0937 x2 = 0.0528 x1 = -0.0580

Therefore, the third column of the inverse coefficient matrix is: -0.0580 0.0528 0.0937 Therefore, the inverse coefficient matrix is: A-1 = -0.0739 -0.0237 -0.0172 Since: x = A-1 (b) x = 0.6517 4.3166 9.5620 Therefore: a b c = 0.6517 = 4.3166 = 9.5620 -0.0026 -0.1794 -0.0185 -0.0580 0.0528 0.0937

Drill: Solve the same system of linear algebraic equations. Answer: d = 0.5000 1.0000 -1.2500 0 2.0000 -1.7500 0 0 0.2500

d =

d =

A-1 = 0.5000 -0.2500 -1.2500 a = 2

1.5000 0.2500 -1.7500

0.5000 0.2500 0.2500

b = -1

c = 3

Part III Curve Fitting

Data is often given for discrete values along a continuum. However, you may require estimates at points between the discrete values. This chapter describes techniques to fit curves to such data in order to obtain intermediate estimates.

Chapter 1 Least-Squares Regression

Where the data exhibits a significant degree of error or noise, the strategy is to derive a single curve that represents the general trend of the data. Because any individual data point may be incorrect, we make no effort to intersect every point. Rather, the curve is designed to follow the pattern of the points taken as a group.

Linear Regression
The simplest example of a least-squares approximation is fitting a straight line to a set of paired observations. Suppose we have m observations (xi,yi), the following are the governing equations: a0 (x0) a0 (x1) a0 (x2) . . . a0 (xn) + + + a1 (x1) a1 (x2) a1 (x3) . . . a1 (xn+1) + a2 (x2) + . . + an (xn+0) = (x0 y) + a2 (x3) + . . + an (xn+1) = (x1 y) + a2 (x4) + . . + an (xn+2) = (x2 y) . . eqn. 1 . + a2 (xn+2) + . . + an (x2n) = (xn y) + . . . + an xn

y = a0x0 + a1 x1 + a2 x2

eqn. 2

where n is the order of equation, i.e., n=1 for a straight line, n=2 for a parabola, etc. Now, suppose we have the following observations of the water temperature as a function of time: 1:00am 2:00am 3:00am 4:00am 5:00am 6:00am 7:00am 0.5C 2.5C 2.0C 4.0C 3.5C 6.0C 5.5C

Shown below is the plot of the observations.

We observe that the data has a general trend with the shape of a straight line. We now fit a straight line to the data using regression. From eqn. 1 we have: a0 (x0) a0 x + + a1 x a1 (x2) = y = (xy)

Solving the equations, we have: a0 = 0.0714 a1 = 0.8393

Substituting into eqn. 2, we have the equation of the straight line: y = 0.0714 + 0.8393x Shown below is the straight line plotted over the data.

According to the fitted straight line, the estimated temperature at 5:30am is 4.6875C. The freezing point was reached at around x = -0.0851 or shortly before midnight at 11:54:54pm. The standard error of the estimates is the square root of the quotient: Sr /(m-(n+1)) where Sr is the sum of the squares of the difference between the estimates and the observations: Observations 0.5C 2.5C 2.0C 4.0C 3.5C 6.0C 5.5C Estimates 0.9107C 1.7500C 2.5893C 3.4286C 4.2679C 5.1071C 5.9464C

The standard deviation of the observations about the mean can be computed as the square root of the quotient: St /(m-1) where St is the sum of the squares of the difference between the observations and their mean. The standard error of the estimates is 0.7734 and the standard deviation of the observations is 1.9457. Now, because the standard error is less than the standard deviation, the linear regression model has merit. The extent of the improvement of our model is quantified by the coefficient of determination: (St-Sr)/St The coefficient of determination is 0.8683. This means that 86.83% of the original uncertainty has been explained by our linear regression model. Drill: What if the data does not have a general trend with the shape of a straight line? What would be the results if, despite this fact, we try to fit a straight line using leastsquares regression? 0000hrs 0001hrs 0002hrs 0003hrs 0004hrs 0005hrs 2.1C 7.7C 13.6C 27.2C 40.9C 61.1C

Answer:

y = -3.7238 + 11.6629x

According to the fitted straight line, the estimated temperature at 5:30am is 60.4219C. The freezing point was reached at around x = 0.3193 or shortly after midnight at 12:19:09am. The standard error of the estimates is quite high at 5.7664 and the standard deviation of the observations is 22.4205. Now, because the standard error is less than the standard deviation, the linear regression model has merit. The coefficient of determination is 0.9471. This means that 94.71% of the original uncertainty has been explained by our linear regression model.

Polynomial Regression
We can fit a parabola (n=2) into the data using the same equations eqn. 1 and eqn. 2: a0 (x0) a0 (x1) a0 (x2) + + + a1 (x1) a1 (x2) a1 (x3) + a2 (x2) = (x0 y) + a2 (x3) = (x1 y) + a2 (x4) = (x2 y)

y = a0x0 + a1 x1 + a2 x2 In solving for the constants a0, a1 and a2 you may use any method learned in Part II or you may use a programmable calculator, if you have one. Most programmable calculators can only handle up to three (3) unknowns. Thus, for n=3, where there are four (4) unknowns, you will have to use the methods learned in Part II. We use the same data as reproduced below: 1:00am 2:00am 3:00am 4:00am 5:00am 6:00am 7:00am 0.5C 2.5C 2.0C 4.0C 3.5C 6.0C 5.5C

The parabola is given by: y = -0.2857 + 1.0774x - 0.0298x2 Shown below is the graph of the parabola plotted over the data.

According to the fitted straight line, the estimated temperature at 5:30am is 4.7396C. The freezing point was reached at around x = 0.2672 or shortly after midnight at 12:16:02am. The standard error of the estimates is 0.8539 and the standard deviation of the observations as computed earlier is 1.9457. Now, because the standard error is less than the standard deviation, the parabolic regression model has merit. The coefficient of determination is 0.8716. This means that 87.16% of the original uncertainty has been explained by our parabolic regression model. Based on the error of estimates, our linear regression model (0.7734) is the better fit as compared to our parabolic regression model (0.8539). Based on the coefficient of determination, our parabolic regression model (87.16%) is still the better fit as compared to our linear regression model (86.83%). Shown below are the graphs of both the linear (red) and parabolic (blue) fits plotted over the data.

Through visual inspection we find that the two models are very much alike within the range of the observations. We also find that they have two common estimates and these are computed as: 1.7500C at 2:00am and 5.1071C at 6:00am. Drill: Using the data from the previous drill, fit a parabola using least-squares regression. Compare the results with that of the linear model of the previous drill. Answer: y = 2.4786 + 2.3593x + 1.8607x2

According to the fitted parabola, the estimated temperature at 5:30am is 71.7413C. The parabola does not touch the x-axis. The standard error of the estimates is 1.1175 and the standard deviation of the observations as computed earlier is 22.4205. Now, because the standard error is less than the standard deviation, the parabolic regression model has merit. The coefficient of determination is 0.9985. Based on the error of estimates, our parabolic regression model (1.1175) is the better fit as compared to our linear regression model (5.7664). Based on the coefficient of determination, our parabolic regression model (99.85%) is still the better fit as compared to our linear regression model (94.71%). Below are graphs of the linear (red) and parabolic (blue) fits plotted over the data.

The parabola is a better fit than the linear model. They have two common estimates and these are computed as: 45.3515C at 4:12:28am and 5.5152C at 12:47:32am.

Chapter 2 Nonlinear Models

Linear regression provides a powerful technique for fitting a best line to a data. However, it is predicated on the fact that the relationship between the dependent and independent variables is linear. This is not always the case and so transformations can be used to express the data in a form that is compatible with linear regression.

Exponential Model
This model is used in many fields of engineering to characterize quantities that increase (b>0) or decrease (b<0) at a rate that is directly proportional to their own magnitude. This model is expressed by the equation: y = a ebx where a and b are constants. We use the data in the previous drill as reproduced below: 0000hrs 0001hrs 0002hrs 0003hrs 0004hrs 0005hrs 2.1C 7.7C 13.6C 27.2C 40.9C 61.1C eqn. 1

The transformation of eqn. 1 is done by getting its natural logarithm. Then we change variables by letting: y1 = ln(y) Then we have: y1 = a0 + a1x1 If we do linear regression on data (x1,y1 ) we have: a0 = 1.1422 a1 = 0.6444 Reverting to our original variables, we have: a = 3.1337 b = 0.6444 Hence, our exponential model is: y = 3.1337 e0.6444x x1 = x

Shown below is the graph of the exponential model plotted against the data.

Shown below is the same graph (red) but this time the parabolic fit (blue), as computed earlier in the previous drill, is superimposed. How do the exponential and parabolic fits compare?

According to the exponential model, the estimated temperature at 5:30am is 108.4943C. The boiling point is reached at around x = 5.3735 or at 05:22:25am. The first point (0,2.1) is located below both curves since: fexp(0) = 3.1337 fpar(0) = 2.4786

The sum of the squares of the discrepancies can be computed as: Sr = (yi - ye)2 where yi is any of the observations and ye is the corresponding estimate. The error Sr of the exponential fit is 346.3562 while that of the parabolic fit is only 3.7466. Clearly, the parabolic fit is better. Drill: Using the data given below fit an exponential curve and a parabola. Compare. 1 2 3 4 5 Answer: y = 0.3431 e0.6853x y = -0.2000 + 0.4371x + 0.2571x2 0.5 1.7 3.4 5.7 8.4

The table below shows that only the first 3 points are located between the curves. x 1 2 3 4 5 y 0.5000 1.7000 3.4000 5.7000 8.4000 y_exp 0.6808 1.3508 2.6805 5.3188 10.5541 y_par 0.4943 1.7029 3.4257 5.6629 8.4143

The error Sr of the exponential fit is 5.4576 while that of the parabolic fit is only 0.0023. Clearly, the parabolic fit is better.

Power Model
This model has wide applicability in all fields of engineering and is expressed by the equation: y = a xb where a and b are constants. We use the same data in the previous drill as shown below: 1 2 3 4 5 0.5 1.7 3.4 5.7 8.4 eqn. 2

The transformation of eqn. 2 is done by getting its base-10 logarithm. Then we change variables by letting: y1 = log(y) Then we have: y1 = a0 + a1x1 If we do linear regression on data (x1,y1 ) we have: a0 = -0.3002 a1 = 1.7517 Reverting to our original variables, we have: a = 0.5009 b = 1.7517 Hence, our power model is: y = 0.5009 x1.7517 Shown below is the graph of the power model plotted against the data. We observe from the graph that our power model fits our data as accurately as the parabolic model that was computed in the previous drill. However, the error Sr of the power fit is 0.0016 while that of the parabolic fit is, as previously computed, 0.0023. Clearly, the power fit is better. x1 = log(x)

Drill: Compute the power model of the first 9 terms of the Fibonacci sequence. Answer: y = 0.4768 x1.6624

Saturation-Growth-Rate Model
This model is particularly well-suited for characterizing population growth rate under limiting conditions. It represents a non-linear relationship between y and x that levels off or saturates as x increases and is expressed by the equation: y = ax / (x+b) where a and b are constants. We use the data shown below. 7 9 15 25 40 75 100 150 0.29 0.37 0.48 0.65 0.80 0.97 0.99 1.07 eqn. 3

The linearization of eqn. 3 is done by inverting it. We then change variables by letting: y1 = 1/y x1 = 1/x Then we have: y1 = a0 + a1x1 If we do linear regression on data (x1,y1 ) we have: a0 = 0.8127 a1 = 18.0365 Reverting to our original variables, we have: a = 1.2304 b = 22.1927 Hence, our saturation-growth-rate model is: y = 1.2304x / (x+22.1927) Shown below is the graph of the model plotted against the data.

We observe that as x increases the graph tends to saturate at the value of 1. In fact, if we substitute x= to our model we get the exact value of this saturation point: x = 1.2304 The error Sr of our model is 0.0013.

Drill: Compute the saturation-growth-rate and parabolic models of the data shown below. Which model is better? 0.7 0.9 1.5 2.5 4.0 7.5 10.0 15.0 Answer: y = 1.2803x / (x+2.3531) y = 0.2678 + 0.1337x - 0.0048x2 0.29 0.37 0.48 0.65 0.75 0.97 1.09 1.20

The error Sr of the saturation model is 0.0154 while that of the parabolic model is 0.0142. Clearly, the parabolic model is better.

Chapter 3 Multiple Regression

Multiple Linear Regression
A useful extension of linear regression is the case where y is a linear function of two or more variables. Such an equation is particularly useful when fitting experimental data where the variable being studied is often a function of two other variables as in: y = a0 + a1 x1 + a2 x2 + . . . + eqn. 2 ++ an xn = y an xn eqn. 1

The coefficients can be solved using the system: ma0 a0 x1 a0 x2 . . . a0 xn + + + a1 x1 + a2 x2

a1 (x1x1) + a1 (x1x2) + . . . a1 (x1xn) +

a2 (x2x1) ++ a2 (x2x2) ++ . . . a2 (x2xn) ++

an (xnx1) = (x1y) an (xnx2) . . . an (xnxn) = (x2y) . . . = (xny)

where m is the number of observations and n is the number of independent variables. We use the data shown below: x1 0 2 2.5 1 4 7 From eqn. 1 we have: y = a0 + a1 x1 + a2 x2 0 1 2 3 6 2 x2 y 5 10 9 1 3 27

From eqn. 2 we have: 6a0 a0 x1 a0 x2 + + + a1 x1 + a2 x2 = y = (x1y) = (x2y)

a1 (x1x1) + a2 (x2x1) a1 (x1x2) + a2 (x2x2)

Solving the system of equations, we have: a0 = 5.2252 a1 = 3.9232 a2 = -2.9345 Hence, our multiple linear regression model is: y = 5.2252 + 3.9232x1 2.9345x2

The error Sr of our model is 0.6553. Drill: Compute the saturation-growth-rate model of the data shown below. x1 0 2 2.5 1 4 7 Answer: y = 5.3697 + 3.7093x1 3.0082x2 + 0.0672x3 0 1 2 3 6 2 x2 0 1 4 9 16 25 x3 y 5 10 9 1 3 27

The error Sr of our model is 0.3413.

Chapter 4 Interpolation
Where the data is known to be very precise, the basic approach is to fit a curve or a series of curves that pass directly through each of the points. Such data usually originates from tables. Polynomial interpolation consists of determining the unique nth-order polynomial that fits n+1 data points. This polynomial then provides a formula to compute intermediate values.

Newtons Divided-Difference Interpolating Polynomials

Consider the graph shown below:

If we interpolate at x between two known values f(x0) and f(x1), we have: f(x) = f(x0) + f(x1)-f(x0) x1-x0 Let: b0 = f(x0) b1 = f(x1)-f(x0) x1-x0 We then have the 1st -order Newtons interpolating polynomial: f(x) = b0 + b1 (x-x0) eqn. 2 eqn. 3 (x-x0) eqn. 1

Now, let us estimate the natural logarithm of 2 using Newtons linear interpolation between the values 1 and 6, decreasing the upper value by 0.5 until it reaches 4. Results: x0 1.0000 1.0000 1.0000 1.0000 1.0000 x1 6.0000 5.5000 5.0000 4.5000 4.0000 f(x) 0.3584 0.3788 0.4024 0.4297 0.4621 error(%) 48.3007 45.3460 41.9518 38.0021 33.3333

Last estimate is 0.4621. As against the actual value, the error is 33.3333%. Drill: Estimate sin(/4) using Newtons linear interpolation between the values 0 and /2, decreasing the upper value by 0.1 until the error is below 10%. Answer: 0.6437 @ 8.9697% Drill: Derive equation #1. Previously we derived the 1st -order Newtons interpolating polynomial: f(x) = b0
nd

b1 (x-x0)

However, if three data points are available, then we can expand this equation to get the 2 -order Newtons interpolating polynomial: f(x) = b0 + b1 (x-x0) + b2 (x-x0)(x-x1)

If we let x=x2 and using eqn. 2 and eqn. 3, we will get the equation for b2 : f(x2) f(x1) x2 - x1 b2 = x2 - x0 Let us estimate the natural logarithm of 2 using Newtons quadratic interpolation across 1, 4 and 6, decreasing the upper values by 0.2 until error < 10%. The results are: x0 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 x1 4.0000 3.8000 3.6000 3.4000 3.2000 3.0000 2.8000 x2 6.0000 5.8000 5.6000 5.4000 5.2000 5.0000 4.8000 f(x) 0.5658 0.5763 0.5872 0.5986 0.6104 0.6228 0.6357 error(%) 18.3659 16.8582 15.2866 13.6475 11.9371 10.1519 8.2882 x1 x0 eqn. 4 f(x1) f(x0)

Our last estimate is 0.6357 with an error of 8.2882%.

Drill: Estimate sin(/4) using Newtons quadratic interpolation across the values 0, /3 and /2, decreasing the two upper values by 0.1 until the error is below 1%. Answer: 0.7105 @ 0.4778% Drill: Derive equation #4. From our two equations: f(x) = b0 f(x) = b0 + + b1 (x-x0) b1 (x-x0) linear quadratic

b2 (x-x0)(x-x1)

we can write the general form of Newtons interpolating polynomial as: fn(x) = b0 + b1(x-x0) + . . + bn(x-x0)(x-x1). .(x-xn-1)

where n is the order of the interpolating polynomial. Hence, for the 3rd-order Newtons interpolating polynomial, we have: f3(x) = b0 + b1(x-x0) + b2 (x-x0)(x-x1) + b3(x-x0)(x-x1)(x-x2) If we let x=x3 and using equations 2, 3 and 4, we can derive the equation for b3. However, this derivation is very long and is prone to errors. Hence, a short-cut is in order. This short-cut is in the form of a finite divided difference. Our first finite divided difference is equation #3 which involves x0 and x1, and which we designate as F[0,1]: b1 = F[1,0] = f(x1)-f(x0) x1-x0 If we analyze equation #4, it looks like: b2 = F[2,1,0] = F[2,1] - F[1,0] x2 - x0 If we follow the trend, we have: b3 = F[3,2,1,0] = F[3,2,1] - F[2,1,0] x3 - x0 From equation #6, we have: F[3,2,1] = F[3,2] - F[2,1] x3 x1 eqn. 6 eqn. 5

From equation #5, we have: F[3,2] = f(x3)-f(x2) x3-x2 Hence we can now have an equation of b3 using a finite divided difference. Now, let us estimate the natural logarithm of 2 using 3rd-order Newtons interpolation across the values 1, 4, 5 and 6, decreasing all upper values by 0.2 until the error is below 10%. We have the results: x0 1.0000 1.0000 x1 4.0000 3.8000 x2 5.0000 4.8000 x3 6.0000 5.8000 f(x) 0.6288 0.6359 error(%) 9.2879 8.2635

The answer is 0.6288 with an error of 9.2879%. Drill: Estimate sin(/4) using 3rd-order Newtons interpolation across the values 0, /3, /2 and , decreasing all upper values by 0.1 until the error is below 1%. Answer: 0.7093 @ 0.3086%

Lagrange Interpolating Polynomials

The Lagrange interpolating polynomial is simply a reformulation of the Newton polynomial that avoids the computation of divided differences. The 1st-order Newton interpolating polynomial, as given by equation #1 above, is reproduced below: f1(x) = f(x0) + f(x1) - f(x0) (x-x0) x1-x0 If we try to get rid of the divided difference: f(x1) - f(x0) x1-x0 We can transform it, hence: f1(x) = f(x0) + f(x1)(x-x0) x1-x0 f(x0)(x-x0) x1-x0

f1(x) = f(x0) + (x-x0) f(x1) + (x-x0) f(x0) x1-x0 x0-x1

f1(x) = f(x0)[ 1 +

x-x0 ] x0-x1

(x-x0) f(x1) x1-x0 eqn. 7

f1(x) = (x-x1) f(x0) + (x-x0) f(x1) (x0-x1) (x1-x0)

The 2nd-order Newtons interpolating polynomial, previously derived above, is reproduced below: f2(x) = b0 + b1 (x-x0) + b2 (x-x0)(x-x1)

Using equations #2 and #3, we have: f2(x) = f(x0) + f(x1) - f(x0) (x-x0) x1-x0 Using equation #4 and after some manipulations, we will have: f2(x) = (x-x1)(x-x2) f(x0) (x0-x1)(x0-x2) (x-x0)(x-x1) (x2-x0)(x2-x1) f(x2) eqn. 8 + (x-x0)(x-x2) f(x1) (x1-x0)(x1-x2) + + b2 (x-x0)(x-x1)

Drill: Give the complete and detailed derivation of equation #8. We can generalize our results from equations #7 and #8 to derive a concise representation of the nth-order Lagrange interpolating polynomial: fn(x) = [Li(x)f(xi)] eqn. 9

where n is the order of the polynomial and the index of the summation (i) takes values from 0 to n and where: Li(x) = (x-xj)/(xi-xj) eqn. 10

where the index of the product (j) takes values from 0 to n and ji. Notice the divisor in equation #10. Always bear in mind that where j is equal to i, you must skip this factor, else you will have a division by zero. Drill: Based on equations #9 and #10, write the 3rd -order Lagrange interpolating polynomial. Now, let us do the interpolation exercises again, this time using Lagrange interpolating polynomials.

Let us estimate the natural logarithm of 2 using Lagrange linear interpolation between the values 1 and 6, decreasing the upper value by 0.5 until it reaches 4. We have the same results as obtained above using Newtons linear interpolating polynomial: x0 1.0 1.0 1.0 1.0 1.0 x1 6.0 5.5 5.0 4.5 4.0 f(x) 0.3584 0.3788 0.4024 0.4297 0.4621 error(%) 48.3007 45.3460 41.9518 38.0021 33.3333 L0 0.8000 0.7778 0.7500 0.7143 0.6667 L1 0.2000 0.2222 0.2500 0.2857 0.3333

Our last estimate is 0.4621 with an error of 33.3333% with L0=0.6667 L1=0.3333 Drill: Estimate sin(/4) using Lagrange linear interpolation between the values 0 and /2, decreasing the upper value by 0.1 until the error is below 10%. Answer: 0.6437 @ 8.9697% with L0=0.2665 L1=0.7335 Now, let us estimate the natural logarithm of 2 using Lagrange quadratic interpolation across the values 1, 4 and 6, decreasing the two upper values by 0.2 until the error is below 10%. We have the same results as obtained above using Newtons quadratic interpolating polynomial: x0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 x1 4.0 3.8 3.6 3.4 3.2 3.0 2.8 x2 6.0 5.8 5.6 5.4 5.2 5.0 4.8 f(x) 0.5658 0.5763 0.5872 0.5986 0.6104 0.6228 0.6357 error(%) 18.3659 16.8582 15.2866 13.6475 11.9371 10.1519 8.2882 L0 0.5333 0.5089 0.4816 0.4508 0.4156 0.3750 0.3275 L1 0.6667 0.6786 0.6923 0.7083 0.7273 0.7500 0.7778 L2 -0.2000 -0.1875 -0.1739 -0.1591 -0.1429 -0.1250 -0.1053

Our last estimate is 0.6357 with an error of 8.2882% L0=0.3275 L1=0.7778 L2=-0.1053 Drill: Estimate sin(/4) using Lagrange quadratic interpolation across the values 0, /3 and /2, decreasing the two upper values by 0.1 until the error is below 1%. Answer: 0.7105 @ 0.4778% L0=0.0312 L1=1.0365 L2=-0.0676 Now, let us estimate the natural logarithm of 2 using 3rd-order Lagrange interpolation across the values 1, 4, 5 and 6, decreasing all upper values by 0.2 until the error is below 10%. We have the results: x1 4.0 3.8 x2 5.0 4.8 x3 6.0 5.8 f(x) error L0 L1 L2 L3

0.6288 9.2879 0.4000 2.0000 -2.0000 0.6000 0.6359 8.2635 0.3750 1.9000 -1.8000 0.5250

The answer is 0.6288 with an error of 9.2879% with L0=0.4000 L1=2.0000 L2=-2.0000 L3=0.6000 Drill: Estimate sin(/4) using 3rd-order Lagrange interpolation across the values 0, /3, /2 and , decreasing all upper values by 0.1 until the error is below 1%. Answer: 0.7093 @ 0.3086% with L0 = 0.0228 L1 = 1.0671 L2 = -0.0928 L3 = 0.0029

Spline Interpolation
In the previous sections, nth-order polynomials were used to interpolate between n+1 data points. However, there are cases where these functions can lead to erroneous results. An alternative approach is to apply lower-order polynomials to subsets of data points. Such connecting polynomials are called spline functions. The simplest connection between two points is a straight line. The 1st-order splines for a group of ordered data points can be defined as a set of linear functions, one linear function per interval: f1(x) = m1x + b1 f2(x) = m2x + b2 . . . fn(x) = mnx + bn x0 x x1 x1 x x2

xn-1 x xn

where mi is the slope of the straight line connecting the points (xi,yi) and (xi-1,yi-1) and bi is the corresponding y-intercept. These equations can be used to evaluate the functions at any point between x0 and xn by first locating the interval within which the point lies. Then the appropriate equation is used to determine the function value within the interval. Suppose we fit the data shown below with 1st-order splines. x 3.0 4.5 7.0 9.0 f(x) 2.5 1.0 2.5 0.5

We have the following linear splines: f1(x) = -x + 5.5 f2(x) = 0.6x 1.7 f3(x) = -x + 9.5 3.0 x 4.5 4.5 x 7.0 7.0 x 9.0

The graph of the three linear splines is shown below.

To estimate the value at x=5, we use the second linear spline since its interval contains x=5. Hence f(x=5) = 1.3000. Drill: Fit the data shown below with 1st-order splines and estimate the value of the function at x=5. x 3.0 4.5 7.0 9.0 Answer: f1(x) = x - 2.0 f2(x) = -0.8x + 6.1 f3(x) = x - 6.5 f(x=5) = 2.1000 3.0 x 4.5 4.5 x 7.0 7.0 x 9.0 f(x) 1.0 2.5 0.5 2.5

A visual inspection of the two foregoing graphs indicates that the primary disadvantage of linear or 1st -order splines is that they are not smooth. In essence, at the data points where two splines meet (called the knot), the slope changes abruptly. In formal terms, the first derivate of the function is discontinuous at these points. This deficiency is overcome by using higher-order polynomial splines that ensure smoothness at the knots. This smoothness is achieved by equating derivatives at the knots. Let us now learn how to formulate 2nd-order or quadratic splines. These splines have continuous first derivates at the knots. The objective is to derive a 2nd-order polynomial for each interval between data points: f1(x) = a1x2 + b1x + c1 f2(x) = a2x2 + b2x + c2 . . . fn(x) = anx2 + bnx + cn x0 x x1 x1 x x2

xn-1 x xn

For n+1 data points there are n intervals and, consequently, 3n unknown constants to evaluate. Therefore, 3n equations or conditions are required to evaluate the unknowns. These are: 1. The function values must be equal at the interior knots. ai-1 xi-1 ai xi-1
2 2

+ +

bi-1 xi-1 bi xi-1

+ +

ci-1 ci

= =

f(xi-1) f(xi-1)

for i = 2 (first interior knot) to n (last interior knot) Each equation provides n-1 conditions, for a total of 2n-2 conditions.

2. The first and last functions must pass through the end points. a1 x02 an xn2 + + b1 x0 bn xn + + c1 cn = = f(x0) f(xn) first function through first point last function through last point

This adds two conditions for a total of 2n-2+2 = 2n conditions. 3. The first derivatives at the interior knots must be equal. This ensures that the transition of the splines through the interior knots will be smooth. ai-1 xi-1 2 ai xi-1 2 2ai-1 xi-1 2ai xi-1 Hence: 2ai-1 xi-1 + bi-1 = 2ai xi-1 + bi + + + + bi-1 xi-1 bi xi-1 bi-1 bi + + ci-1 ci left side of interior knot right side of interior knot first derivative of left side first derivative of right side

for i = 2 (first interior knot) to n (last interior knot) This equation provides n-1 conditions, for a total of 2n+(n-1) = 3n-1 conditions. 4. Assume that the second derivative is zero at the first point. aix 2 + 2aix + 2ai 2a1 = 0 Hence: bi x bi + ci original function first derivative second derivative zero at the first point

a1 = 0

This brings our number of conditions to a total of 3n-1+1= 3n. Let us now fit the data with quadratic splines by going through each of the four sets of conditions: 1. The function values must be equal at the interior knots. a1x12 a2x12 a2x22 a3x22 (eqn. 1) (eqn. 2) (eqn. 3) (eqn. 4) + + + + b1x1 b2x1 b2x2 b3x2 + + + + + + + + c1 c2 c2 c3 4.5b1 4.5b2 7b2 7b3 = = = = + + + + f(x1) f(x1) f(x2) f(x2) c1 c2 c2 c3 = = = = 1 1 2.5 2.5

20.25a1 20.25a2 49a2 49a3

2. The first and last functions must pass through the end points. a1 x02 a3 x32 + + b1 x0 b3 x3 + + + + 3b1 9b3 c1 c3 = = + + f(x0) f(x3) c1 = c3 = 2.5 0.5

(eqn. 5) 9a1 (eqn. 6) 81a3

3. The first derivatives at the interior knots must be equal. 2a1 x1 2a2 x2 + + b1 b2 + + = = b1 b2 2a2 x1 2a3 x2 = = + + b2 b3 + + b2 b3

(eqn. 7) 9a1 (eqn. 8) 14a2

9a2 14a3

4. Assume that the second derivative is zero at the first point. (eqn. 9) a1 = 0 Collecting all the 3n = 9 equations we have the table below. a1 20.25 0 0 0 9 0 9 0 1 b1 4.5 0 0 0 3 0 1 0 0 1 0 0 0 1 0 0 0 0 c1 a2 0 20.25 49 0 0 0 -9 14 0 b2 0 4.5 7 0 0 0 -1 1 0 0 1 1 0 0 0 0 0 0 c2 a3 0 0 0 49 0 81 0 -14 0 b3 0 0 0 7 0 9 0 -1 0 0 0 0 1 0 1 0 0 0 c3 = = = = = = = = = 1 1 2.5 2.5 2.5 0.5 0 0 0

To solve for the 9 unknowns we can use any of the methods learned in Part II (or we may opt for the fast and dirty solution). We have the following results: a1 = 0 a2 = 0.6400 a3 = -1.6000 b1 = -1.0000 b2 = -6.7600 b3 = 24.6000 c1 = 5.5000 c2 = 18.4600 c3 = -91.3000

Therefore, our quadratic splines are: f1(x) = -x + 5.5 f2(x) = 0.64x2 6.7600x + 18.46 f3(x) = -1.6000x2 + 24.6000x 91.3000 f(x=5) = 0.6600 3.0 x 4.5 4.5 x 7.0 7.0 x 9.0

Notice that our first spline is not really a 2nd- order polynomial. The graph of the three splines is shown below on the right side.

Compare our new graph (quadratic splines) with the old one (linear splines reproduced above on the left side). Notice the smooth function transition between the interior knots. However, notice also that there are two shortcomings that detract from the new fit: 1. The straight line connecting the first two points. 2. The spline for the last interval seems to swing too high. Cubic or 3rd-order splines do not exhibit these shortcomings and as a consequence are usually better methods for spline interpolation. Drill: Fit the data from the last drill with 2nd -order splines and estimate the value of the function at x=5. Answer: f1(x) = x 2 f2(x) = -0.7200x2 + 7.4800x 16.5800 f3(x) = 1.8000x2 - 27.8000x + 106.9000 f(x=5) = 2.8200 3.0 x 4.5 4.5 x 7.0 7.0 x 9.0

Let us now design cubic splines by deriving a 3rd order polynomial for each interval: f1(x) = a1x3 + b1x2 + c1x + d1 f2(x) = a2x3 + b2x2 + c2x + d2 . . . fn(x) = anx3 + bnx2 + cnx + dn x0 x x1 x1 x x2

xn-1 x xn

For n+1 data points there are n intervals and, consequently, 4n unknown constants to evaluate. Therefore, 4n equations or conditions are required to evaluate the unknowns. These are (please note that #1, #2 and #3 are identical to those from quadratic splines, except for the order of the function): 1. The function values must be equal at the interior knots. ai-1 xi-1 3 + bi-1 xi-1 ai xi-1 3 + bi xi-1
2 2

+ ci-1 xi-1 + ci xi-1

+ +

di-1 di

= =

f(xi-1) f(xi-1)

For i = 2 (first interior knot) to n (last interior knot). Each equation provides n-1 conditions, for a total of 2n-2 conditions. 2. The first and last functions must pass through the end points. a1 x03 + b1 x02 + c1 x0 + d1 = f(x0) first function through first point an xn3 + bn xn2 + cn xn + dn = f(xn) first function through last point This adds two conditions for a total of 2n-2+2 = 2n conditions.

3. The first derivatives at the interior knots must be equal. This ensures that the transition of the splines through the interior knots will be smooth. ai-1 xi-1 ai xi-1
3 3

+ bi-1 xi-1 + bi xi-1

2 2

2 2

+ ci-1 xi-1 + ci xi-1 + ci-1 + ci

+ +

di-1 left side of interior knot di right side of interior knot first derivative of left side first derivative of right side

3ai-1 xi-1 3ai xi-1 Hence: 3ai-1 xi-1

+ 2bi-1 xi-1 + 2bi xi-1

+ 2bi-1 xi-1 + ci-1

3ai xi-1

+ 2bi xi-1 + ci

for i = 2 (first interior knot) to n (last interior knot) This equation provides n-1 conditions, for a total of 2n+(n-1) = 3n-1. 4. The second derivatives at the interior knots must be equal. This ensures that the transition of the splines through the interior knots will be smooth. Using the results from #3 above, we have: 6ai-1 xi-1 6ai xi-1 Hence: + 2bi-1 + 2bi + 2bi-1 = second derivative of left side second derivative of right side 6ai xi-1 + 2bi

6ai-1 xi-1

for i = 2 (first interior knot) to n (last interior knot) This equation provides n-1 conditions, for a total of 3n-1+n-1) = 4n-2. 5. The second derivatives at the end knots are zero. Using the second derivative results from #4, we have: 6ai-1 xi-1 6ai xi-1 + 2bi-1 + 2bi second derivative of left side second derivative of right side

At the end knots we have: 6a1 x0 6an xn Hence, we have: 6a1 x0 6an xn + + 2b1 2bn = 0 = 0 + + 2b1 2bn second derivative at first knot second derivative at last knot

This provides two additional conditions, bringing our total to 4n-2+2 = 4n.

Let us now fit the data with cubic splines by going through each of the five sets of conditions: 1. The function values must be equal at the interior knots. a1 a2 a2 a3 x1 x1 x2 x2
3 3 3 3

+ + + +

b1 b2 b2 b3

x1 x1 x2 x2

2 2 2 2

+ + + +

c1 c2 c2 c3 + + + +

x1 x1 x2 x2

+ + + +

d1 d2 d2 d3 + + + + 4.5c1 4.5c2 7c2 7c3

= = = = + + + +

f(x1) f(x1) f(x2) f(x2) d1 d2 d2 d3 = = = = 1 1 2.5 2.5

(eqn. 01) (eqn. 02) (eqn. 03) (eqn. 04)

91.1250a1 91.1250a2 343a2 343a3

20.25b1 20.25b2 49b2 49b3

2. The first and last functions must pass through the end points. a1 x03 + b1 x02 + c1 x0 + d1 = f(x0) a3 x33 + b3 x32 + c3 x3 + d3 = f(x3) (eqn. 05) 27a1 (eqn. 06) 729a3 + 9b1 + 3c1 + d1 = 2.5 + 81b3 + 9c3 + d3 = 0.5

3. The first derivatives at the interior knots must be equal. 3a1 x1 3a2 x2
2 2

+ +

2b1 x1 + 2b2 x2 +

c1 c2 + +

= =

3a2 x1 3a3 x2 c1 c2 = =

2 2

+ 2b2 x1 + + 2b3 x2 + + +

c2 c3 c2 c3

(eqn. 07) 60.75a1 + (eqn. 08) 147a2 +

9b1 14b2

60.75a2 147a3

9b2 + 14b3 +

4. The second derivatives at the interior knots must be equal. 6a1 x1 6a2 x2 + + 2b1 2b2 + + = = 2b1 2b2 6a2 x1 6a3 x2 = = + 2b2 + 2b3 27a2 42a3 + + 2b2 2b3

(eqn. 09) 27a1 (eqn. 10) 42a2

5. The second derivatives at the end knots are zero. 6a1 x0 6a3 x3 + + 2b1 2b3 + + = 0 = 0 2b1 2b3 = 0 = 0

(eqn. 11) 18a1 (eqn. 12) 54a3

We have the following results: a1 = 0.1866 a2 = -0.2141 a3 = 0.1278 b1 = -1.6791 b2 = 3.7305 b3 = -3.4494 c1 = 3.6175 c2 = -20.7256 c3 = 29.5338 d1 = 1.7221 d2 = 38.2367 d3 = -79.0354

Therefore, our cubic splines are: f1(x) = 0.1866x3 1.6791x2 + 3.6175x + 1.7221 3.0 x 4.5 f2(x) = -0.2141x3 + 3.7305x2 20.7256x + 38.2367 4.5 x 7.0 f3(x) = 0.1278x3 - 3.4494x2 + 29.5338x - 79.0354 7.0 x 9.0 f(x=5) = 1.1029 Notice that all of our functions are 3rd-order polynomials. Their graph is shown below on the right side.

Compare our new graph (cubic splines) with the old one (quadratic splines reproduced above on the left side). Notice the disappearance of the straight line connecting the first two points and of the high-swinging spline in the last interval.

Drill: Fit the data shown below with cubic splines. x 3.0 4.5 6.0 7.0 9.0 Answer: This time, there are 4 intervals. Hence, n=4. Also, there will be 4n=16 unknowns to evaluate. 1. The function values must be equal at the interior knots. a1 a2 a2 a3 a3 a4 x1 x1 x2 x2 x3 x3
3 3 3 3 3 3

f(x) 1.0 2.5 2.0 0.5 2.5

+ + + + + +

b1 b2 b2 b3 b3 b4

x1 2 x1 2 x2 2 x2 2 x3 2 x3 2

+ + + + + +

c1 c2 c2 c3 c3 c4 + + + + + +

x1 x1 x2 x2 x3 x3

+ + + + + +

d1 d2 d2 d3 d3 d4 + + + + + + 4.5c1 4.5c2 6c2 6c3 7c3 7c4

= = = = = = + + + + + +

f(x1) f(x1) f(x2) f(x2) f(x3) f(x3) d1 d2 d2 d3 d3 d4 = = = = = = 2.5 2.5 2.0 2.0 0.5 0.5

(eqn. 01) (eqn. 02) (eqn. 03) (eqn. 04) (eqn. 05) (eqn. 06)

91.1250a1 91.1250a2 216a2 216a3 343a3 343a4

20.25b1 20.25b2 36b2 36b3 49b3 49b4

2. The first and last functions must pass through the end points. a1 x03 + b1 x02 + c1 x0 + d1 = f(x0) a4 x43 + b4 x42 + c4 x4 + d4 = f(x4) (eqn. 07) 27a1 + 9b1 + 3c1 + d1 (eqn. 08) 729a4 + 81b4 + 9c4 + d4 = 1 = 2.5

3. The first derivatives at the interior knots must be equal. 3a1 x1 3a2 x2 3a3 x3
2 2 2

+ 2b1 x1 + c1 + 2b2 x2 + c2 + 2b3 x3 + c3

= = =

3a2 x1 3a3 x2 3a4 x3 = = =

2 2 2

+ 2b2 x1 + c2 + 2b3 x2 + c3 + 2b4 x3 + c4

(eqn. 09) 60.75a1 + 9b1 + c1 (eqn. 10) 108a2 + 2b2 + c2 (eqn. 11) 147a3 + 14b3 + c3

60.75a2 + 9b2 + c2 108a3 + 12b3 + c3 147a4 + 14b4 + c4

4. The second derivatives at the interior knots must be equal. 6a1 x1 6a2 x2 6a3 x3 + + + 2b1 2b2 2b3 + + + = = = 2b1 2b2 2b3 6a2 x1 6a3 x2 6a4 x3 = = = + 2b2 + 2b3 + 2b4 27a2 36a3 42a4 + 2b2 + 2b3 + 2b4

(eqn. 12) 27a1 (eqn. 13) 36a2 (eqn. 14) 42a3

5. The second derivatives at the end knots are zero. 6a1 x0 6a4 x4 + + 2b1 2b4 + + = 0 = 0 2b1 2b4 = 0 = 0

(eqn. 15) 18a1 (eqn. 16) 54a4

f1(x) = -0.1108x3 + 0.9969x2 - 1.7414x + 0.2430 for 3.0 x 4.5 f2(x) = -0.0388x3 + 0.0250x2 + 2.6320x 6.3172 for 4.5 x 6.0 f3(x) = 0.6784x3 - 12.8833x2 + 79.8320x - 159.7173 for 6.0 x 7.0 f4(x) = -0.2270x3 + 6.1297x2 53.2591x + 150.8286 for 7.0 x 9.0

Chapter 5 Fourier Approximation

Engineers often deal with systems that oscillate or vibrate. As might be expected, trigonometric functions play a fundamental role in modeling such problem contexts. Fourier approximation represents a systematic framework for using trigonometric series for this purpose.

Sinusoidal Functions
A sinusoid is any waveform that can be described as a sine or cosine. There is no clear-cut convention for choosing either function, and in any case the results will be identical. A sinusoid f(t) can be represented generally as a function of time as: f(t) = A + A0 cos(0t + ) eqn. 1

The mean value A sets the average height above the abscissa. The amplitude C specifies the height of the oscillation. The angular frequency 0 characterizes how often the cycles occur. Finally, the phase angle or phase shift parameterizes the extent to which the sinusoid is shifted horizontally. Although equation #1 is an adequate mathematical characterization of a sinusoid, it is awkward to work with from the standpoint of curve fitting because the phase shift is included in the argument of the cosine function. This deficiency can be overcome by invoking a trigonometric identity. After substituting it into the equation and collecting terms we have: f(t) = A + B cos(0t) + C sin(0t) where: B = A0 cos() C = -A0 sin() eqn. 2

Drill: Give the derivation of equation #2. We can treat equation #2 as another example of the general model from the multiple linear regression discussion of Part III Chapter 3: y = a0 + a1 x1 + a2 x2 + . . . + an xn

for n=2, we have: y = a0 + a1 x1 + a2 x2 Equating equations#2 and #3, we have: A = a0 x1 = cos(0t) B = a1 x2 = sin(0t) C = a2 eqn. 3

We then use multiple linear regression to solve for the coefficients.

Let us now fit a sinusoid with 0=4.1890 to the data given below: t 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 1.20 1.35 f(t) 2.2000 1.5950 1.0310 0.7220 0.7860 1.2000 1.8050 2.3690 2.6780 2.6140 A = 1.7000 B = 0.5001 C = -0.8661

After multiple linear regression: Hence, our sinusoidal fit is:

f(t) = 1.7000 + 0.5001 cos(4.1890t) 0.8661 sin(4.1890t) or alternatively from equation #1: f(t) = 1.7000 + 1.0001 cos(4.1890t + 1.0471) or: f(t) = 1.7000 + 1.0001 sin(4.1890t + 2.6179) The error Sr of our sinusoidal model is 0. Shown below is the graph of the sinusoid plotted against the data.

Drill: Fit a sinusoid (0=4) to the data given below: t 0.45 0.60 0.75 0.90 1.05 1.20 1.35 1.50 1.65 1.80 f(t) 0.7000 0.8911 1.3911 2.0092 2.5092 2.7000 2.5088 2.0087 1.3906 0.8908

f(t) = 1.6533 + 0.0876 cos(4t) 1.0189 sin(4t) f(t) = 1.6533 + 1.0227 cos(4t + 1.4850) f(t) = 1.6533 + 1.0227 sin(4t + 3.0558) The error Sr of our sinusoidal model is 0.0067.

Continuous Fourier Series

Fourier showed that an arbitrary periodic function can be represented by an infinite series of sinusoids of harmonically related frequencies. For a function with period T, a continuous Fourier Series can be written as: f(t) = a0 + a1 cos(0t) + b1 sin(0t) + a2 cos(20t) + b2 sin(20t) + . . . or more concisely: f(t) = a0 +

[a

cos(k0t) + bk sin(k0t)

where 0 = 2/T is called the fundamental frequency, its constant multiples 20, 30, etc., are called harmonics and k = 1, 2, 3, . . . The coefficients can be computed via:

ak = 2 T
and

f(t) cos(k0t) dt

bk = 2 T
and

f(t) sin(k t) dt
0

a0 = 1 T

f(t) dt

where k = 1, 2, 3, . . . and the integration is over the period T. Use the continuous Fourier Series to approximate the rectangular wave function shown below.

Analyzing the rectangular wave, we have: f(t) = -1 1 -1 , -T/2 < t < -T/4 , -T/4 < t < T/4 , T/4 < t < T/2

We have:

ak = (2/T)

[-(1/k )sin(k t) + (1/k )sin(k t) (1/k )sin(k t)]

0 0 0 0 0 0

Since 0 = 2/T:

ak = -(1/k)sin(k0t) + (1/k)sin(k0t) (1/k)sin(k0t) ak = -(1/k)[sin(-k/2) - sin(-k)] +(1/k)[sin(k/2) - sin(-k/2)] -(1/k)[sin(k) - sin(k/2)] ak = -(1/k)[-sin(k/2) + sin(k)] +(1/k)[sin(k/2) + sin(k/2)] -(1/k)[sin(k) - sin(k/2)] ak = (1/k)sin(k/2) - (1/k)sin(k) (1/k)sin(k/2) + (1/k)sin(k/2) (-1/k)sin(k) + (1/k)sin(k/2) ak = (4/k)sin(k/2) - (2/k)sin(k) a1 a2 a3 a4 a5 = = = = = (4/)sin(/2) - (2/)sin() (2/)sin() - (1/)sin(2) (4/3)sin(3/2) - (2/3)sin(3) (1/)sin(2) - (1/2)sin(4) (4/5)sin(5/2) - (2/5)sin(5) = = = = = 4/ 0 -4/3 0 4/5

Hence: ak = 4/k -4/k 0

for k = 1, 5, 9, . . . for k = 3, 7, 11, . . . for k = 2, 4, 6, . . .

We have:

bk = (2/T)

[(1/k )cos(k t) - (1/k )cos(k t) + (1/k )cos(k t)]

0 0 0 0 0 0

Since 0 = 2/T:

bk = (1/k)cos(k0t) - (1/k)cos(k0t) + (1/k)cos(k0t) bk = (1/k)[cos(-k/2) - cos(-k)] -(1/k)[cos(k/2) - cos(-k/2)] +(1/k)[cos(k) - cos(k/2)] bk = (1/k)[cos(k/2) - cos(k)] -(1/k)[cos(k/2) - cos(k/2)] +(1/k)[cos(k) - cos(k/2)] bk = (1/k)cos(k/2) - (1/k)cos(k) -(1/k)cos(k/2) + (1/k)cos(k/2) (1/k)cos(k) - (1/k)cos(k/2)
We have:

bk =

a0 = -(1/T)[(-T/4) (-T/2)] +(1/T)[(T/4) (-T/4)] -(1/T)[(T/2) (T/4)] a0 = -(1/T)[(-T/4) + (T/2)] +(1/T)[(T/4) + (T/4)] -(1/T)[(T/2) (T/4)] a0 = -(1/T)(T/4) +(1/T)(T/2) -(1/T)(T/4) a0 = -1/4 + 1/2 1/4 a0 = 0

Therefore, the Fourier Series approximation is:

f(t) = 4

cos(0t) 4 cos(30t) + 4 cos(50t) 3 5 cos(70t) + . . .

- 4 7

Shown below is the graph of the Fourier Series approximation (up to the fourth term) plotted against the rectangular wave.

Drill: Use a continuous Fourier Series to approximate the saw-tooth wave (T=2) shown below.

Answer: Analyzing the wave, we have f(t) = 0.5t Hence, , 0 < t < 2

ak = (2/T)

0.5t cos(k0t) dt

ak = 0.5
Since:

t cos(k0t) dt

ak = 0.5 cos(k0t) + 0.5t sin(k0t) (k0)2 k0

Since 0 = 2/T = :

ak = 0.5 cos(kt) + 0.5t sin(kt) k (k)2 ak = 0.5 cos(k2) + (k)2 ak = 0.5 (k)2 sin(k2) k + sin(k2) k 0.5 cos(0) (k)2

sin(k2) k ak = 0

0.5 (k)2

ak =

Next, we have:

bk = 2 T

0.5t sin(k0t) dt

bk = 0.52
Since:

t sin(k t) dt
0

bk = 0.5 sin(k0t) - 0.5t cos(k0t) (k0)2 k0

Since 0 = 2/T = :

bk = 0.5 sin(kt) - 0.5t cos(kt) (k)2 k

bk = 0.5 sin(k2) - cos(k2) 0.5 sin(0) (k)2 k (k)2 bk = 0 - cos(k2) 0 k bk = - cos(k2) k

Then we have:

bk =

-1/(k)

a0 = 1 T

0.5t dt

a0 = 0.25 t2/2 a0 = 0.125 t2 = 0.125(4-0)

Therefore, the Fourier Series approximation is: f(t) = 0.5 1 sin(0t) 1 sin(20t)

a0 =

0.5

- 1 sin(30t) 1

2
sin(40t) - . . .

Shown below is the graph of the Fourier Series approximation (up to the fifth term) plotted against the saw-tooth wave.

Part IV Numerical Differentiation and Integration

Calculus is the mathematics of change. Because engineers must continuously deal with systems and processes that change, calculus is an essential tool of our profession. Standing at the heart of calculus are the related mathematical concepts of differentiation and integration.

Chapter 1 Newton-Cotes Integration Formulas

The Newton-Cotes formulas are the most common numerical integration schemes. They are based on the strategy of replacing a complicated function or tabulated data with some approximating function (a polynomial) that is easy to integrate. For example, in Figure 1-a, a first-order polynomial (trapezoidal rule) is used as an approximation. In Figure 1-b, a second-order polynomial (Simpsons rules) is used for the same purpose. The integral can also be approximated using a series of polynomials applied piece-wise (i.e., multiple application) to the function or data over segments of constant length. In Figure 2, three straight lines are used to approximate the integral. Higher-order polynomials can be utilized for the same purpose.

Figure 1

Figure 2

Closed and open forms of the Newton-Cotes formulas are available. The closed forms are those where the data points at the beginning and end of the limits of integration are known. The open forms have integration limits that extend beyond the range of the data. The Open Newton-cotes formulas are not generally used for definite integration. However, they are utilized for evaluating improper integrals and for the solution of ordinary differential equations.

Trapezoidal Rule
The trapezoidal rule is the first of the Newton-Cotes closed integration formulas. It corresponds to the case where the approximating polynomial used is first-order as shown in Figure 3.

Figure 3 Recall that in Part III, Chapter 4 equation #1, the line can be represented as: f1(x) = f(x0) + f(x1)-f(x0) (x-x0) x1-x0 The area under this straight line (i.e., the integral of f1(x)) is an estimate of the integral of f(x) between the limits x0 and x1 : I

f1(x)

eqn. 1

Drill: Derive the trapezoidal rule formula from equation #1. Let us now use the trapezoidal rule to numerically integrate: f(x) = from 0 to 0.8. 400x5 - 900x4 + 675x3 - 200x2 + 25x + 0.2

The exact value of the integral is I = 1.6405. Using equation #1, the estimate of the integral is I 0.1728, with an error of 89.4668 %. The large error is evident from the fact that the trapezoid covers only a very small area under the curve, neglecting a significant area above the line, as shown in Figure 4.

Figure 4

Drill: Use the trapezoidal rule to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Answer: I 25.1327 error = 28.4635%

Multiple-Application of Trapezoidal Rule

One way to improve the accuracy of the trapezoidal rule is to divide the integration interval into a number of segments and apply the method to each segment. The areas of individual segments can then be added to yield the integral for the entire interval.

Let us now use multiple application of the trapezoidal rule to numerically integrate: f(x) = 400x5 - 900x4 + 675x3 - 200x2 + 25x + 0.2

from 0 to 0.8. Let us use the intervals [0 0.4] and [0.4 0.8]. The exact value of the integral for the first interval is I1 = 0.5632. The estimate of this integral is I1 0.5312. The exact value of the integral for the second interval is I2 = 1.0773. The estimate of this integral is I2 0.5376. Combining the two estimates we have I 1.0688, with an error of 34.85%. Compare this error with that of the single application method which is 89.4668 %.

Drill: Use multiple application of the trapezoidal rule to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Use the intervals [0 /3] and [/3 ]. Answer: The exact value of the integral for the first interval is I1 = 10.8776. The estimate of this integral is I1 10.6448. The exact value of the integral for the second interval is I2 = 24.2552. The estimate of this integral is I2 21.2897. Combining the two estimates we have I 31.9345, with an error of 9.10%. Compare this error with that of the single application method which is 28.4635%.

Aside from multiple application of the trapezoidal rule, another way to obtain a more accurate estimate of an integral is to use higher-order polynomials to connect the points. The formulas that result from taking the integrals under these polynomials are called Simpsons Rules.

Simpsons 1/3 Rule

This rule results when a second-order interpolating polynomial is used to approximate an integral. Recall that in Part III, Chapter 4 equation #8, a second-order Lagrange Interpolating polynomial is given by: f2(x) = (x-x1)(x-x2) f(x0) (x0-x1)(x0-x2) (x-x0)(x-x1) (x2-x0)(x2-x1) The integral of this second-order interpolating polynomial, evaluated from x0 to x2, gives what is called Simpsons 1/3 Rule: (where x1 is midway between x0 and x2) f(x2) + (x-x0)(x-x2) f(x1) (x1-x0)(x1-x2) +

f2(x)

eqn. 2

Drill: Derive Simpsons 1/3 Rule formula from equation #2. Let us now use Simpsons 1/3 Rule to numerically integrate the same function from 0 to 0.8. The exact value of the integral is I = 1.6405. Using equation #2, the estimate of the integral is I 1.3675, with an error of 16.6450%.

Drill: Use Simpsons 1/3 rule to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Answer: I 35.6047 error = 1.3434%

Multiple-Application of Simpsons 1/3 Rule

Simpsons 1/3 rule can be improved by dividing the integration interval into a number of segments of equal width. Let us now use multiple application of Simpsons 1/3 rule to numerically integrate the same function from 0 to 0.8. Let us use the intervals [0 0.4] and [0.4 0.8]. The estimate of the integral for the first interval is I1 0.5205. For the second interval the estimate is I2 1.1029. Combining the two estimates we have I 1.6235, with an error of 1.04%.

Drill: Use multiple application of Simpsons 1/3 rule to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Use the intervals [0 /2] and [/2 ]. Answer: The estimate of the integral for the first interval is I1 17.5778. For the second interval the estimate is I2 17.5778. Combining the two estimates we have I 35.1555, with an error of 0.065%.

Simpsons 3/8 Rule

This rule results when a third-order interpolating polynomial is used to approximate an integral. Recall that in Part III, Chapter 4 equations #9 and #10, a third-order Lagrange interpolating polynomial is given by: f3(x) = (x-x1)(x-x2)(x-x3)f(x0) + (x-x0)(x-x2)(x-x3) f(x1) + (x0-x1)(x0-x2)(x0-x3) (x1-x0)(x1-x2)(x1-x3)

(x-x0)(x-x1)(x-x3)f(x2) + (x-x0)(x-x1)(x-x2) f(x3) (x2-x0)(x2-x1)(x2-x3) (x3-x0)(x3-x1)(x3-x2)

The integral of this third-order interpolating polynomial, evaluated from x0 to x3, gives what is called Simpsons 3/8 Rule: (where x1 and x2 are equally spaced)

f3(x)

eqn. 3

Drill: Derive Simpsons 3/8 Rule formula from equation #3.

Let us now use Simpsons 3/8 Rule to numerically integrate the same function from 0 to 0.8. The exact value of the integral is I = 1.6405. Using equation #3, the estimate of the integral is I 1.5192 with an error of 7.3978%.

Drill: Use Simpsons 3/8 rule to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Answer: I 35.3354 error = 0.5767%

Combined Application of Simpsons 1/3 and 3/8 Rules

The combination of Simpsons 1/3 and 3/8 rules require application to five (5) segments. The first two segments are for the 1/3 rule and the last three segments are for the 3/8 rule. Let us now use combined application of Simpsons 1/3 and 3/8 rules to numerically integrate the same function from 0 to 0.8. Let us divide the interval into five equally-spaced sub-intervals.

The estimate of the integral for the first two sub-intervals is I1 0.3803. For the last three sub-intervals the estimate is I2 1.2648. Combining the two estimates we have I 1.6451, with an error of 0.28%.

Drill: Use a combination of Simpsons 1/3 and 3/8 rules to numerically integrate f(x) = 8 + 5sin(x) from 0 to . Divide the interval into five equallyspaced sub-intervals. The estimate of the integral for the first two sub-intervals is Answer: I1 13.5111. For the last three sub-intervals the estimate is I2 21.6388. Combining the two estimates we have I 35.1499, with an error of 0.05%.

It must be stressed that, in practice, higher-order formulas are rarely used. Simpsons rules are sufficient for most applications. Accuracy can be improved by using the multipleapplication version rather than opting for the higher-point formulas. The tables below summarize the results and compare the errors from the methods discussed above.

Table1 Comparison of Errors

method

error %

example function Trapezoidal Rule Multiple Trapezoidal Rule Simpsons 1/3 Rule Multiple Simpsons 1/3 Rule Simpsons 3/8 Rule Combined Simpsons 1/3 & 3/8 Rules 89.47 34.85 16.65 1.04 7.40 0.28

drill function 28.46 9.10 1.34 0.07 0.58 0.05

Table 2 Estimates of the Integrals

method

example function

drill function

h* Trapezoidal Rule Multiple Trapezoidal Rule Simpsons 1/3 Rule Multiple Simpsons 1/3 Rule Simpsons 3/8 Rule Combined Simpsons 1/3 & 3/8 Rules
*

I 0.1728 1.0688 1.3675 1.6235 1.5192 1.6451

h*

0.8 0.4 0.4 0.2 4/15 0.16

25.1327 /3 2/3 31.9345 /2 35.6047 /4 35.1555 /3 35.3354 /5 35.1499

h is the length of segment

Integration with Unequal Segments

All the formulas discussed above are based on equally spaced data points. However, there are many situations where this assumption does not hold and we must deal with unequal-sized segments. For these cases, one method is to apply the trapezoidal rule to each segment and sum the results. Note that this is the same approach used for the multipleapplication trapezoidal rule. In fact, the drill problem under multiple-application trapezoidal rule uses segments of unequal lengths, i.e., [0 /3] and [/3 ]. Use the trapezoidal rule to determine the integral of f(x) evaluated from 0 to 2/5.

x 0.00 0.12 0.22 0.32 0.36 0.40

f(x) 0.2000 1.3097 1.3052 1.7434 2.0749 2.4560

There are five segments and the integral for each is listed in the table below:

segment 0.00 0.12 0.22 0.32 0.36 0.12 0.22 0.32 0.36 0.40

integral 0.0906 0.1307 0.1524 0.0764 0.0906

total

0.5407

Drill: Use the trapezoidal rule to determine the integral of f(x) evaluated from 2/5 to 4/5. Answer:

x 0.40 0.44 0.54 0.64 0.70 0.80

f(x) 2.4560 2.8430 3.5073 3.1819 2.3630 0.2320

There are five segments and the integral for each is listed in the table below:

segment 0.40 0.44 0.54 0.64 0.70 0.44 0.54 0.64 0.70 0.80

integral 0.1060 0.3175 0.3345 0.1663 0.1298

total

1.0541

Using the data given below, let us compute the integral of f(x) using Simpsons rules for those segments where they are appropriate.

x 0.00 0.12 0.22 0.32 0.36 0.40 0.44

f(x) 0.2000 1.3097 1.3052 1.7434 2.0749 2.4560 2.8430

The segments and the methods to be used are listed below.

segment 1 2 3 4 5 6 0.00 0.12 0.22 0.32 0.36 0.40 0.12 0.22 0.32 0.36 0.40 0.44

length 0.12 0.10 0.10 0.04 0.04 0.04

method trapezoid Simpsons 1/3 Simpsons 3/8

Segments 1 and 2 cannot be solved using a Simpsons rule because they are of unequal lengths. Hence, segment 1 will be solved using trapezoidal rule. Segments 2 and 3 are of equal lengths and, hence, can be solved using Simpsons 1/3 rule. Segments 4 and 5 are of equal lengths and, hence, can also be solved using Simpsons 1/3 rule. However, since segment 6 is also of equal length as segments 4 and 5, and since Simpsons 3/8 rule results to less errors, we will group segments 4, 5 and 6 and solve them using Simpsons 3/8 rule. The results are tabulated below.

segment 1 2 3 4 5 6 0.00 0.12 0.22 0.32 0.36 0.40 0.12 0.22 0.32 0.36 0.40 0.44

integral 0.0906 0.2758 0.2727

total

0.6391

Drill: Using the data given below, compute the integral of f(x) using Simpsons rules for those segments where they are appropriate.

x 0.44 0.54 0.64 0.70 0.80

f(x) 2.8430 3.5073 3.1819 2.3630 0.2320

The segments and the methods to be used are listed below.

segment 1 2 3 4 0.44 0.54 0.64 0.70 0.54 0.64 0.70 0.80

length 0.10 0.10 0.06 0.10

method

Simpsons 1/3 trapezoid trapezoid

segment 1 2 3 4 0.44 0.54 0.64 0.70 0.54 0.64 0.70 0.80

integral

0.6685 0.1663 0.1298

total

0.9646

Chapter 2 Integration of Equations

Functions to be integrated numerically will typically be of two forms: a table of values or an equation. There are two techniques designed to analyze cases where the function equation is given. The first is based on Richardsons Extrapolation, which is a method for combining two numerical integral estimates in order to obtain a third, more accurate value. The second method is called Gauss Quadrature.

Richardsons Extrapolation (Romberg Integration)

Richardsons extrapolation is designed to obviate the shortcomings of multipleapplication trapezoidal rule. It is quite similar to the techniques of Chapter 1 in the sense that it is based on successive application of the trapezoidal rule. However, through mathematical manipulations, superior results are attained for less effort. Romberg integration is a computational algorithm for implementing Richardsons extrapolation in a highly efficient manner. The estimate and error associated with a multiple-application trapezoidal rule can be represented generally as: I = I(h) + E(h) eqn. 1

where I is the exact value of the integral, I(h) is the approximation from an n-segment application of the trapezoidal rule with segment length h, and E(h) is the truncation error. If we make two separate estimates using segment lengths h1 and h2 and have exact values for the error: I(h1) + E(h1) = I(h2) + E(h2) The ratio of the two errors can be approximated by: E(h1) E(h2) Hence, E(h1) E(h2) h12 / h22 Substituting into equation #2, we have: I(h1) + [E(h2) h12 / h22)] = I(h2) + E(h2) Hence, E(h2)[(h12/h22) 1] = I(h2) - I(h1) h 12 h22 eqn. 2

E(h2)

I(h2) - I(h1) [(h12/h22) 1]

Substituting into equation #1, we have Richardsons extrapolation formula: I I(h2) + I(h2) - I(h1) [(h1/h2)
4 2

eqn. 3

1]

Let us now numerically compute the integral of the function f(x) = 400x5 900x + 675x3 - 200x2 + 25x + 0.2, evaluated from 0 to 0.8, using multipleapplication trapezoidal rule of 1, 2, 4 and 8 segments. The answers are summarized in the table below. Table 3

segments 1 2 4 8

h 0.8 0.4 0.2 0.1

I 0.1728 1.0688 1.4848 1.6008

error% 89.5 34.9 09.5 02.4

Using Table 3 and Richardsons extrapolation we can have improved estimates of the integral of the function f(x) = 400x5 - 900x4 + 675x3 - 200x2 + 25x + 0.2, evaluated from 0 to 0.8. The results are summarized in the table below. Table 4

h1 0.8 0.4 0.2

h2 0.4 0.2 0.1

I 1.3675 1.6235 1.6395

error% 16.64 01.04 00.07

Drill: Use Table 4 to do another iteration of Richardsons extrapolation to get more improved estimates of the integral. Answer: The results are summarized in the table below.

h1 0.8 0.4

h2 0.2 0.1

I 1.6406 1.6406

error% 0.00 0.00

Drill: Do a final iteration of Richardsons extrapolation to get the best estimate of the integral. Answer: The results are summarized in the table below.

h1 0.8

h2 0.1

I 1.6406

error% 0.00

The formulations of Richardsons extrapolation can be expressed in a general form that is well-suited for computer implementation. This is called the Romberg Integration:

Ij,k = 4k-1 Ij+1,k-1 Ij,k-1 4k-1 - 1

eqn. 4

where Ij+1,k-1 and Ij,k-1 are the more and less accurate integrals, respectively, Ij,k is the improved integral and k signifies the level of integration. Hence, k=1 corresponds to the original trapezoidal rule estimates, k=2 corresponds to the estimates using the first iteration of the Romberg Integral, k=3 corresponds to the estimates using the second iteration of the Romberg Integral, and so on. Drill: Use Romberg Integration to numerically compute the integral of the function evaluated from 0 to 3. f(x) = ex sin(x) dx 1 + x2 Answer:

segments 1 2 4 8

h 3.000 1.500 0.750 0.375

I 0.4252 2.2759 2.7438 2.8478

h1 3.00 1.50 0.75

h2 1.500 0.750 0.375

I 2.8928 2.8998 2.8825

h1 3.00 1.50

h2 0.750 0.375

I 2.9003 2.8813

h1 3.0

h2 0.375

I 2.8810

Gauss Quadrature (Gauss-Legendre Formulas)

A characteristic of most of the formulas in Chapter 1 is that the integral estimates are based on evenly spaced function values. Consequently, the location of the base points used in these equations is predetermined or fixed. Suppose that the constraint of fixed base points is removed and we are free to evaluate the area under a straight line joining any two points on the curve. By positioning these points wisely, we could define a straight line that would balance the positive and negative errors. Hence, we would arrive at an improved estimate of the integral. The Gauss quadrature is one technique to implement such a strategy. The particular Gauss quadrature formulas described in this section are called Gauss-Legendre formulas. The object of the Gauss quadrature is to determine the coefficients of an equation of the form: I c0f(x0) + c1f(x1) eqn. 1

where c0 and c1 are the unknown coefficients. Also, since the function arguments x0 and x1 are not fixed at the endpoints, but are unknowns, we have a total of four unknowns that must be evaluated. Consequently, we require four conditions to determine them exactly. We can obtain two of these conditions by assuming that equation #1 fits the integral of a constant (y=1) and a linear (y=x) function exactly:

c0f(x0) + c1f(x1) =

(1) dx

c0f(x0) + c1f(x1) =

(x) dx

To arrive at the two other conditions, we merely extend this reasoning by assuming that it also fits the integral of a parabolic (y=x2) and a cubic (y=x3) function.

c0f(x0) + c1f(x1) =

(x2) dx

c0f(x0) + c1f(x1) =

(x3) dx

Assuming the limits of integration are -1 and 1, we have: c0f(x0) + c1f(x1) = c0f(x0) + c1f(x1) = c0f(x0) + c1f(x1) = c0f(x0) + c1f(x1) = or, we have: c0f(x0) + c1f(x1) = 2 c0f(x0) + c1f(x1) = 0 c0f(x0) + c1f(x1) = 2/3 c0f(x0) + c1f(x1) = 0 eqn. 2 eqn. 3 eqn. 4 eqn. 5 dx x dx x2 dx x3 dx = 2 = 0 = 2/3 = 0

Inspecting equation #2, since the function is a constant function y=1, we have: f(x0) = f(x1) = 1 Hence, c0 + c1 = 2 eqn. 6

Inspecting equation #3, since the function is a linear function y=x, we have: f(x0) = x0 Hence, c0x0 + c1x1 = 0 eqn. 7 f(x1) = x1

Inspecting equation #4, since the function is a parabolic function y=x2, we have: f(x0) = (x0)2 Hence, c0(x0)2 + c1(x1)2 = 2/3 eqn. 8 f(x1) = (x1)2

Inspecting equation #5, since the function is a parabolic function y=x3, we have: f(x0) = (x0)3 Hence, c0(x0)3 + c1(x1)3 = 0 In summary we have: eqn. 9 f(x1) = (x1)3

c0 c0x0 c0(x0)2 c0(x0)3

+ + + +

c1 c1x1 c1(x1)2 c1(x1)3

= = = =

2 0 2/3 0

eqn. 10 eqn. 11 eqn. 12 eqn. 13

Combining equations #10 and #11, we have:

x0 = -(2-c0)x1/c0

eqn. 14

Substituting equations #14 and #10 into equation #12, we have:

(x1)2 = -c0 /[3(c0-2)]

eqn. 15

Substituting equations #14 and #10 into equation #13, we have:

c0 = 1
Hence, from equation #10:

c1 = 1
From equation #15:

x1 = (1/3)1/2
From equation #14:

x0 = -(1/3)1/2
Finally, from equation #1, we have the Two-Point Gauss-Legendre Formula:

I f[-(1/3)1/2] + f[(1/3)1/2]

eqn. 16

Notice that the integration limits we used are from -1 to 1. This was done to simplify the mathematics and to make the formulation as general as possible. A simple change of variable can be used to translate other limits of i9ntegration into this form. This is

accomplished by assuming that a new variable z is related to the original variable x in a linear fashion: x = a0 + a1 z eqn. 17

If the lower limit, x=a, corresponds to z=-1, these values can be substituted into equation #17 to yield: a = a0 - a1 eqn. 18

Similarly, if the upper limit, x=b, corresponds to z=1, then: b = a0 + a1 Adding equations #18 and #19 yields: a0 = (b + a)/2 Subtracting equation #19 from #18 yields: a1 = (b - a)/2 Substituting equations #20 and #21 into #17 yields: x = (b+a) + (b-a)z 2 Let us now use the two-point Gauss-Legendre formula to numerically calculate the integral of the function f(x) = 400x5 - 900x4 + 675x3 - 200x2 + 25x + 0.2, evaluated from 0 to 0.8. Before integrating the function, we must perform a change of variable so that the limits are from -1 to 1. Using equation #22, where a=0 and b=0.8, we have: x = 0.4 + 0.4z dx = 0.4dz eqn. 22 eqn. 21 eqn. 20 eqn. 19

We then substitute these equations into the original function to give the transformation: f(z) = [400(0.4+0.4z)5 900(0.4+0.4z)4 + 675(0.4+0.4z)3 200(0.4+0.4z)2 + 25(0.4+0.4z) + 0.2] [0.4dz] Evaluating this function using equation #16, we have: f[-(1/3)1/2] = 0.5167 Hence: I 1.8226 error = 11.10% f[(1/3)1/2] = 1.3058

Beyond the two-point formula, higher-point versions can be developed in the general form: I c0f(x0) + c1f(x1) + . . . + cn-1f(xn-1) where n is the number of points. The following table shows the weighing factors c and function arguments x used in Gauss-Legendre formulas. Table 5

points

weighing factors

function arguments

c0 = 1 c1 = 1

x0 = -0.57735 x1 = 0.57735

c0 = 0.55556 c1 = 0.88889 c2 = 0.55556

x0 = -0.77460 x1 = 0 x2 = 0.77460

c0 c1 c2 c3

= = = =

0.34785 0.65215 0.65215 0.34785

x0 x1 x2 x3

= -0.86114 = -0.33998 = 0.33998 = 0.86114

c0 c1 c2 c3 c4

= = = = =

0.23693 0.47863 0.56889 0.47863 0.23693

x0 x1 x2 x3 x4

= -0.90618 = -0.53847 = 0 = 0.53847 = 0.90618

c0 c1 c2 c3 c4 c5

= = = = = =

0.17132 0.36076 0.46791 0.46791 0.36076 0.17132

x0 x1 x2 x3 x4 x5

= -0.93247 = -0.66121 = -0.23862 = 0.23862 = 0.66121 = 0.93247

Drill: Use the three-point Gauss-Legendre formula to numerically calculate the integral of the same function, evaluated from 0 to 0.8. Answer: I 0.2813 + 0.8732 + 0.4860 1.6405

error = 0 %

Drill: Use Gauss-Legendre formulas up to six points to numerically calculate the integral of the function f(x), evaluated from 0 to 10, where g=9.8, c=12.5, and m=68.1.

f(x) = (gm/c)[1 e-cx/m]

Answer: The table below summarizes the results.

points

error %

2 3 4 5 6

290.0145 289.4404 289.4353 289.4364 289.4323

0.20 0.00 0.00 0.00 0.00

Improper Integrals (Extended Midpoint Rule)

In this section we will focus on one type of improper integral that is, one with a lower limit of or an upper limit of +. Such integrals usually can be evaluated by making a change of variable that transforms the infinite range to one that is finite. The following identity serves this purpose and works for any function that decreases towards zero at least as fast as 1/x2 as x approaches infinity:

eqn. 22 for ab>0. Therefore, it can only be used when a is positive and b is or when a is - and b is negative. For cases where the limits are from - to a positive value or from a negative value to , the integral can be implemented in two steps. Fore example:

eqn. 23 where -A is chosen as a sufficiently large negative value so that the function has begun to approach zero asymptotically at least as fast as 1/x2. After the integral has been divided into two parts, the first can be evaluated with equation #22 and the second with a NewtonCotes closed formula such as Simpsons 1/3 rule.

Problem Using equations #22 and #23 and A=-2, solve for N(1).

Using equation #23, we can express N(1) as:

The first integral can be evaluated by applying equation #22 to give:

This integral can be solved using the extended midpoint rule:

f(x) dx = h[f(m )+f(m )+. . .]

0 1

evaluated from q to r, where m0, m1, etc. are the midpoints between the interval [q r]. The first midpoint is h/2 away from q, the second is 3h/2 away, the next is 5h/2 away and so on towards the direction of r. Hence, if we choose h=1/8, then:

midpoint

f(m)

-0.4375 -0.3125 -0.1875 -0.0625 total

0.3833 0.0612 0.0000 0.0000 0.4445

Therefore, the first integral is evaluated as: I1 [1/(2)1/2] * h * 0.4445 0.0222 The second integral can be evaluated using Simpsons 1/3 rule (h=0.5) to give:

segment

-2.0000 -1.0000 0

-1.5000 -0.5000 0.5000 total

-1.0000 0 1.0000

0.3401 0.8561 0.8561 2.0523

I2 [1/(2)1/2] * 2.0523 0.8187 Hence, N(1) 0.0222 + 0.8187 0.8409

Drill: Solve for N(1) using h=1/16 for the first integral and h=0.25 for the second. Answer:

midpoint

f(m)

-0.4688 -0.4063 -0.3438 -0.2813 -0.2188 -0.1563 -0.0938 -0.0313 total

0.4676 0.2929 0.1230 0.0227 0.0006 0.0000 0.0000 0.0000 0.9068

I1 0.0226

segment

-2.0000 -1.5000 -1.0000 -0.5000 0 0.5000

-1.7500 -1.2500 -0.7500 -0.2500 0.2500 0.7500 total

-1.5000 -1.0000 -0.5000 0.0000 0.5000 1.0000

0.1104 0.2302 0.3757 0.4800 0.4800 0.3757 2.0519

I2 0.8186

N(1) 0.8412

Chapter 3 Numerical Differentiation

Taylors Theorem states that if the function f and its first n+1 derivatives are continuous on an interval containing a and x, then the value of the function at x is given by: f(x) = f(a) (x-a)0 + f(a) (x-a)1 + f(a) (x-a)2 + . . . 0! 1! 2!

+ f(n)(a) (x-a)n + Rn n! The Taylors Series expansion is obtained by replacing x with xi+1 and a with xi: f(xi+1) = f(xi)(xi+1 - xi)0 + f(xi)(xi+1 - xi)1 + . . . 0! 1!

+ f(n)(xi) (xi+1 - xi)n + Rn n!

This theorem and its associated formula, the Taylors Series, is of great value in the study of numerical methods. In essence, the Taylor series provides a means to predict a function value at one point in terms of the function value and its derivatives at another point.

First Forward, Backward and Central Divided Difference Approximations

It is convenient to simplify the series by defining a step size h = xi+1 - xi. Then the Taylors Series expansion will become: f(xi+1) = f(xi) h0 0! + f(xi) h1 1! + . . . + f(n)(xi) hn n! + Rn

Let us truncate the series after the first derivative term (i.e., n=1): f(xi+1) = f(xi) h0 0! + f(xi) h1 1! + R1

f(xi+1) = f(xi) + f(xi) h + R1 This can now be solved for the derivative in the form of a finite divided difference: f(xi) = f(xi+1) - f(xi) h R1 h

The first part of the equation gives us the formula to approximate the first derivative while the second part gives us the formula to compute the truncation error associated with the approximation of the first derivative. Hence, the first forward divided difference approximation is given by: f(xi) f(xi+1) - f(xi) h eqn. 1

Similarly, the first backward divided difference approximation is given by: f(xi) f(xi) - f(xi-1) h eqn. 2

Lastly, the first central divided difference approximation is given by: f(xi) f(xi+1) - f(xi-1) 2h eqn. 3

Let us now use the first forward, backward and central divided difference equations to estimate the first derivative of: f(x) = -0.1x4 - 0.15x3 - 0.5x2 - 0.25x + 1.2 at x=0.5 using a step size h=0.5. The results are tabulated below:

derivative forward backward central -1.4500 -0.5500 -1.0000

error % 58.90 39.73 9.59

Drill: What are the effects on the results if the step size is halved? Answer: The results are tabulated below:

derivative forward backward central -1.1547 -0.7141 -0.9344

error % 26.54 (halved) 21.75 (halved) 2.40 (quartered)

Second Forward, Backward and Central Divided Difference Approximations

The Taylors Series expansion can be written up to the second derivative: f(xi+1) = f(xi) h0 0! + f(xi) h1 1! + f(xi) h2 2! eqn. 4 + R2

f(xi+1) = f(xi) + f(xi) h + 0.5 f(xi) h2

Also, it can be written for f(xi+2) in terms of f(xi) up to the second derivative: f(xi+2) = f(xi) + f(xi) 2h + f(xi) (2h)2 2 Equation #4 can be substituted into equation #5 to yield the second forward divided difference approximation: f(xi) = f(xi) - 2f(xi+1) + f(xi+2) h
2

eqn. 5

eqn. 6

Similar manipulation can be employed to derive the second backward divided difference approximation:

f(xi) = f(xi) - 2f(xi-1) + f(xi-2) h2 Lastly, we have the second central divided difference approximation: f(xi) = f(xi+1) - 2f(xi) + f(xi-1) h
2

eqn. 7

eqn. 8

Let us now use the second forward, backward and central divided difference equations to estimate the second derivative of: f(x) = -0.1x4 - 0.15x3 - 0.5x2 - 0.25x + 1.2 at x=0.5 using a step size h=0.5. The results are tabulated below:

derivative forward backward central -3.1500 -1.0500 -1.8000

error % 80.00 40.00 2.86

Drill: What are the effects on the results if the step size is halved? Answer: The results are tabulated below:

derivative forward backward central -2.3625 -1.3125 -1.7625

error % 35.00 (halved) 25.00 (halved) 0.71 (quartered)

Richardson Extrapolation
Richardson extrapolation is another way to improve derivative estimates when employing finite divided differences aside from decreasing the step size. The method uses two derivative estimates to compute a third, more accurate approximation. From Chapter 2 (Integration of Equations) above, equation #3, the Richardson extrapolation formula for integration, is reproduced below:

I(h2) +

I(h2) - I(h1) [(h1/h2)2 1]

In a similar fashion, we can write the Richardson extrapolation formula for derivatives as: D D(h2) + D(h2) - D(h1) [(h1/h2)
2

eqn. 9

1]

Let us now use Richardson extrapolation to estimate the first derivative of: f(x) = -0.1x4 - 0.15x3 - 0.5x2 - 0.25x + 1.2 at x=0.5 using step sizes h1=0.5 and h2=0.25. Recall from previous results that:

h1 D(h1) forward backward central -1.4500 -0.5500 -1.0000 error % 58.90 39.73 9.59 D(h2) -1.1547 -0.7141 -0.9344

h2 error % 26.54 21.75 2.40

Using equation #9, we have:

D forward backward central -1.0562 -0.7688 -0.9125

error % 15.75 15.75 0

Observe that in the present case Richardson extrapolation on the central divided difference approximations yields a perfect estimate.

Drill: Use Richardson extrapolation to estimate the second derivative of the same function at x=0.5 using step sizes h1=0.5 and h2=0.25. Answer: The results are tabulated below.

Recall from previous results that:

h1 D(h1) forward backward central And so we have: -3.1500 -1.0500 -1.8000 error % 80.00 40.00 2.86 D(h2) -2.3625 -1.3125 -1.7625

h2 error % 35.00 25.00 0.71

D forward backward central -2.1000 -1.4000 -1.7500

error % 20.00 20.00 0

Romberg Algorithm
As with integration, Chapter 2 (Integration of Equations) above, the Richardson extrapolation can be applied iteratively using the Romberg algorithm until the result falls below an acceptable error criterion. Let us now use the Romberg algorithm to estimate the first derivative of the same function at x=0.5 using step sizes 1.0, 0.5 and 0.25. The results are tabulated below: forward error backward D error 68.5 39.7 21.8 central error 38.4 9.6 2.4

h 1.00 0.50 0.25 1.00 0.50 0.50 0.25 1.00 0.25

-2.2375 -1.4500 -1.1547

145.2 -0.2875 58.9 -0.5500 26.5 -0.7141

-1.2625 -1.0000 -0.9344

-1.1875 -1.0562

30.1 -0.6375 15.8 -0.7688

30.1 15.8

-0.9125 -0.9125

0 0

-1.0475

14.8 -0.7775

14.8

-0.9125

Drill: Use the Romberg algorithm to estimate the second derivative of the same function at x=0.5 using step sizes 1.0, 0.5 and 0.25. Answer: forward error backward D error 40.0 40.0 25.0 central error 11.4 2.9 0.7

h 1.00 0.50 0.25 1.00 0.50 0.50 0.25 1.00 0.25

-5.2500 -3.1500 -2.3625

200.0 -1.0500 80.0 -1.0500 35.0 -1.3125

-1.9500 -1.8000 -1.7625

-2.4500 -2.1000

40.0 -1.0500 20.0 -1.4000

40.0 20.0

-1.7500 -1.7500

0 0

-2.0767

18.7 -1.4233

18.7

-1.7500

Part V Ordinary Differential Equations

A solution of an ordinary differential equation is a specific function of the independent variable and parameters that satisfies the original differential equation.

Chapter 1 Eulers Method

This chapter is devoted to solving ordinary differential equations of the form: dy/dx = f(x,y) An example of a differential equation is: dv/dt = g -(cv/m) eqn. 1

which is a model that relates the acceleration of a falling object, such as a parachutist, to the forces acting on it. The parameter g is the gravitational constant, c is the drag coefficient and m is the mass. Equation #1 is a differential equation because it is written in terms of the differential rate of change (dv/dt) of the variable that we are interested in predicting. However, the exact solution of this equation for the velocity of the falling parachutist cannot be obtained using simple algebraic manipulation. Rather, more advanced techniques must be applied to obtain an exact or analytical solution. For example, if the falling parachutist is initially at rest (v=0 at t=0), calculus can be used to solve the equation for: v(t) = (gm/c) [1 e-ct/m] eqn. 2

Equation #2 is called an analytical or exact solution because it exactly satisfies the original differential equation. Unfortunately, there are many mathematical models that cannot be solved exactly. In many cases, the only alternative is to develop a numerical solution that approximates the exact solution. Numerical methods are those in which the mathematical problem is reformulated so it can be solved by arithmetic operations. This can be illustrated for Newtons second law by realizing that the time rate of change of velocity can be approximated by: dv/dt = v/t = v(ti+1) v(ti) ti+1 ti Equation #3 is called a finite divided difference approximation of the derivative which can be substituted into equation #1 to give: eqn. 3

v(ti+1) v(ti) ti+1 ti

= g

c m

v(ti)

This equation can then be rearranged to yield: v(ti+1) = v(ti) + [g(c/m)v(ti)] (ti+1ti) eqn. 4

Eulers Method
This method uses the differential equation to estimate the slope in the form of the first derivative. It uses the slope at the beginning of the interval as an approximation of the average slope over the whole interval. Notice in equation #4 that the term in square brackets, which is the slope, is the righthand side of the differential equation itself (equation #1). If we are given an initial value for velocity at some time ti, we can easily compute velocity at a later time ti+1. This new value of velocity can in turn be employed to extend the computation to velocity at some other time. Equation #4 can be expressed in general terms as: yi+1 = yi + f(xi,yi) h eqn. 5

This formula is referred to as Eulers method. A new value of y is predicted using the slope f(xi,yi) (equal to the first derivative at the original value of x) to extrapolate linearly over the step size h. Let us now compute the velocity of the falling parachutist prior to opening the chute. Suppose that the parachutist has a mass of 68.1 kg and that the drag coefficient is 12.5 kg/s. Let us employ a step size of 2 s. Some of the results are tabulated below.

t 00 02 04 06 08 10 12

v 00.0000 19.6000 32.0047 39.8555 44.8243 47.9690 49.9592

Drill: After a sufficiently long time, a constant velocity, called the terminal velocity, is reached. This velocity is constant because, eventually, the force of gravity will be in balance with the air resistance. Thus, the net force is zero and acceleration has ceased. Compute this constant.

Answer: 53.3904 m/s

Drill: Use Eulers method (equation #5) to numerically integrate the equation given below from 0 to 4 with a step size of 0.5. The initial condition at x=0 is y=1. dy/dx = -2x3 + 12x2 - 20x + 8.5 Answer: The analytical solution is y = -0.5x4 + 4x3 - 10x2 + 8.5x + 1 x 0.0 0.5 y euler 1.0000 y true 1.0000 3.2188 error % 000.0

slope = f(0,1) = [-2(0)3 + 12(0)2 20(0) + 8.5] slope = f(0,1) = 8.5 y(0.5) = y(0) + f(0,1)0.5 = 1 + (8.5)0.5 = 5.25 x 0.0 0.5 x 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 y euler 1.0000 5.2500 y euler 1.0000 5.2500 5.8750 5.1250 4.5000 4.7500 5.8750 7.1250 7.0000 y true 1.0000 3.2188 y true 1.0000 3.2188 3.0000 2.2188 2.0000 2.7188 4.0000 4.7188 3.0000 error % 000.0 063.1 error % 000.0 063.1 095.8 131.0 125.0 074.7 046.9 051.0 133.3

Chapter 2 Runge-Kutta Methods

A fundamental source of error in Eulers method is that the derivative at the beginning of the interval is assumed to apply across the entire interval. The Runge-Kutta methods are available to help circumvent this shortcoming. These methods achieve the accuracy of a Taylor series approach without requiring the calculation of higher derivatives.

Heuns Method
This method is designed to improve the estimate of the slope. It involves the determination of two derivatives for the interval one at the initial point and another at the end point. The two derivatives are then averaged to obtain an improved estimate of the slope for the entire interval. Recall that in Eulers method, the slope at the beginning of an interval: yi = f(xi,yi) is used to extrapolate linearly to yi+1: y0i+1 = yi + f(xi,yi) h eqn. 7 eqn. 6

For the standard Eulers method we would stop at this point. However, in Heuns method, equation #7 gives us not the final answer but only an intermediate prediction. This is why we have distinguished it with a superscripted 0. Equation #7 is called a predictor equation. It provides an estimate of yi+1 that allows the calculation of an estimated slope at the end of the interval: yi+1 = f(xi+1,y0i+1) eqn. 8

Thus, the two slopes, equations #6 and #8, can be combined to obtain an average slope m for the interval. This average slope is then used to extrapolate linearly from yi to yi+1 using Eulers method: yi+1 = yi + m h eqn. 9

Equation #9 is called a corrector equation. Hence, Heuns method is a predictor-corrector approach. Let us now use Heuns method to integrate the equation given below from 0 to 4 with a step size of 1. The initial condition at x=0 is y=2. y = 4e0.8x 0.5y Before solving the problem numerically, calculus can be used to determine the following analytical solution (you should be able to derive this):

y = (4/1.3)[ e0.8x - e-0.5x] + 2e-0.5x Now, the numerical solution: x 0 1 From equation #7: y01 = y0 + f(x0,y0) h = 2 + f(0,2) (1) y01 = 2 + [4e0.8(0) 0.5(2)](1) = 5 From equation #6: y0 = f(x0,y0) = f(0,2) = 4e0.8(0) 0.5(2) = 3 From equation #8: y1 = f(x1,y01)= f(1,5) = 4e0.8(1) 0.5(5) = 6.4022 Hence: m = (3+6.4022)/2 = 4.7011 Finally, from equation #9: y1 = y0 + m h = 2 + (4.7011)(1) = 6.7011 x 0 1 x 0 1 2 3 4 y heun 02.0000 06.7011 y heun 02.0000 06.7011 16.3198 37.1992 83.3378 y true 02.0000 06.1946 y true 02.0000 06.1946 14.8439 33.6772 75.3390 error % 00.0 08.2 error % 00.0 08.2 09.9 10.5 10.6 y heun 2.0000 y true 2.0000 6.1946 error % 00.0

Notice how the error sometimes grows as the iterations proceed. Such increase can occur, especially for large step sizes, and they prevent us from drawing the general conclusion that an additional iteration will always improve the result. However, for a sufficiently small step size, the iterations should eventually converge on a single value.

Drill: Use Heuns method to solve the previous drill. Answer: x 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 y heun 1.0000 3.4375 3.3750 2.6875 2.5000 3.1875 4.3750 4.9375 3.0000 y true 1.0000 3.2188 3.0000 2.2188 2.0000 2.7188 4.0000 4.7188 3.0000 error % 00.0 06.8 12.5 21.1 25.0 17.2 09.4 04.6 00.0

Improved Polygon Method

This technique uses Eulers method to predict a value of y at the midpoint of the interval: yi+1/2 = yi + f(xi,yi) h/2 Then this predicted value is used to estimate a slope at the midpoint: yi+1/2 = f(xi+1/2,yi+1/2) eqn. 11 eqn. 10

which is assumed to represent a valid approximation of the average slope for the entire interval. This slope is then used to extrapolate linearly from xi to xi+1 using Eulers method: yi+1 = yi + f(xi+1/2,yi+1/2) h eqn. 12

Notice that because yi+1 is not on both sides, the corrector, equation #12, cannot be applied iteratively to improve the solution. This method is superior to Eulers method because it utilizes a slope estimate at the midpoint of the prediction interval. Let us now use the improved polygon method to solve the previous drill. x 0.0 0.5 Using equation #10: yi+1/2 = y0 + f(x0,y0) h/2 = 1 + f(0,1) h/2 = 1 + [-2(0)3 + 12(0)2 20(0) + 8.5] (0.25) = 3.1250 y IPM 1.0000 y true 1.0000 3.2188 error % 00.0

Using equation #11: yi+1/2 = f(xi+1/2,yi+1/2) = f(0.25,3.1250) = -2(0.25)3 + 12(0.25)2 20(0.25) + 8.5 = 4.2188 Using equation #12: yi+1 = 1 + (4.2188) 0.5 = 3.1094 x 0.0 0.5 x 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 y IPM 1.0000 3.1094 y IPM 1.0000 3.1094 2.8125 1.9844 1.7500 2.4844 3.8125 4.6094 3.0000 y true 1.0000 3.2188 y true 1.0000 3.2188 3.0000 2.2188 2.0000 2.7188 4.0000 4.7188 3.0000 error % 00.0 03.4 error % 00.0 03.4 06.3 10.6 12.5 08.6 04.7 02.3 00.0

Ralstons Method
All the methods discussed above and many other variations of the Runge-Kutta methods can be cast in the generalized form of equation #5: yi+1 = yi + (xi,yi,h) h eqn. 13

where (xi,yi,h) is called an increment function which can be interpreted as a representative slope over the interval. The increment function can be written in general form as: = a1k1 + a2k2 + . . . + ankn where n is the order, the as are constants and the ks are: k1 = f(xi,yi) k2 = f(xi+p1h,yi+q11k1h) k3 = f(xi+p2h,yi+q21k1h+q22k2h) . . . and so on. eqn. 14

Notice that the ks are recurrence relationships. That is, k1 appears in the equation for k2, which appears in the equation for k3, and so forth. This recurrence makes RungeKutta methods efficient for computer calculations. Ralston determined that choosing a2=2/3 provides a minimum bound on the truncation error for the second-order Runge-Kutta algorithms. For this version, a1=1/3 and p1=q11=3/4. Hence, from equations #13 and #14, we have: yi+1 = yi + [(1/3)k1 + (2/3)k2] h k1 = f[xi,yi] k2 = f[xi+(3/4)h,yi+(3/4)hk1] Let us now use Ralstons method to solve the previous drill. x 0.0 0.5 Using equation #16: k1 = f(x0,y0) = -2(0)3 + 12(0)2 20(0) + 8.5 = 8.5 Using equation #17: k2 = f(0.3750,4.1875) = -2(0.3750)3 + 12(0.3750)2 20(0.3750) + 8.5 = 2.5820 Using equation #15: yi+1 = 1 + [2.8333 + 1.7213] (0.5) = 3.2773 x 0.0 0.5 x 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 y ralston 1.0000 3.2773 y ralston 1.0000 3.2773 3.1016 2.3477 2.1406 2.8555 4.1172 4.8008 3.0313 y true 1.0000 3.2188 y true 1.0000 3.2188 3.0000 2.2188 2.0000 2.7188 4.0000 4.7188 3.0000 error % 0.00 1.82 error % 0.00 1.82 3.39 5.81 7.03 5.03 2.93 1.74 1.04 y ralston 1.0000 y true 1.0000 3.2188 error % 0.00 eqn. 15 eqn. 16 eqn. 17

Notice how all the second-order Runge-Kutta methods are superior to Eulers method, which is actually a first-order Runge-Kutta method. Drill: Use the following methods to integrate the equation given below from 0 to 4 with a step size of 1. The initial condition at x=0 is y=2. y = 4e0.8x 0.5y 1. Eulers method 2. Improved Polygon method 3. Ralstons method Answer: The results are summarized below. The previous results on Heuns method are reproduced for purpose of comparison. x 0 1 2 3 4 euler 02.0000 05.0000 11.4022 25.5132 56.8493 e 00 19 23 24 25 heun 02.0000 06.7011 16.3198 37.1992 83.3378 e 00 08 10 11 11 IPM 02.0000 06.2173 14.9407 33.9412 75.9686 e 0.0 0.4 0.7 0.8 0.8 ralston 02.0000 06.4423 15.5822 35.4566 79.3962 e 0.0 4.0 5.0 5.3 5.4

Systems of Equations
Many practical problems in engineering and science require the solution of a system of simultaneous differential equations rather than a single equation. Such system may be represented generally as: dy1/dx = f1(x,y1,y2,...,yn) dy2/dx = f2(x,y1,y2,...,yn) . . . dyn/dx = fn(x,y1,y2,...,yn) The solution of such a system requires that n initial conditions be known at the starting value of x. The procedure for solving a system simply involves applying the onestep technique for every equation at each step before proceeding to the next step. Let us solve the following set of differential equations using Eulers method assuming that at x=0, y1=4 and y2=6. Integrate to x=2 with a step size of 0.5. dy1/dx = -0.5y1 dy2/dx = 4 - 0.3y2 0.1y1

x 0.0 0.5 Using equation #5:

y1 4.0000

y2 6.0000

y1(0.5) = 4 + f1(0,4)(0.5) = 4 + [(-0.5)(4)](0.5) y1(0.5) = 3 y2(0.5) = 6 + f2(0,6)(0.5) = 6 + [4-0.3(6)0.1(4)]0.5 y2(0.5) = 6.9000 Notice that in solving for the new value of y2, the old values of y1 and y2 are used. x 0.0 0.5 x 0.0 0.5 1.0 1.5 2.0 y1 4.0000 3.0000 y1 4.0000 3.0000 2.2500 1.6875 1.2656 y2 6.0000 6.9000 y2 6.0000 6.9000 7.7150 8.4453 9.0941

All the methods previously discussed in this Part for single equations can be extended to the system shown above. Drill: Use Heuns, improved polygon and Ralstons methods to solve the same set of differential equations: Answer: heun y1 y2 4.0000 6.0000 3.1250 6.8325 2.4414 7.5900 1.9073 8.2739 1.4901 8.8877 IPM y1 y2 4.0000 6.0000 3.1250 6.8325 2.4414 7.5900 1.9073 8.2739 1.4901 8.8877 ralston y1 y2 4.0000 6.0000 3.1250 6.8325 2.4414 7.5900 1.9073 8.2739 1.4901 8.8877

x 0.0 0.5 1.0 1.5 2.0

Notice that, in this particular case, the results under the three methods are identical. <end> Reference: Numerical Methods for Engineers 2nd ed., S. Chapra and R. Canale McGraw-Hill Book Company 1990