Thamhain, H. J. Project Evaluation and Selection The Engineering Handbook. Ed. Richard C.

Dorf Boca Raton: CRC Press LLC, 2000

1998 by CRC PRESS LLC

189
Project Evaluation and Selection
189.1 Quantitative Approaches to Project Evaluation and Selection 189.2 Qualitative Approaches to Project Evaluation and Selection
Collective Multifunctional Evaluations Net Present Value (NPV) Comparison Return-on-Investment Comparison Pay-Back Period (PBP) Comparison Pacifico and Sobelman Project Ratings Limitations of Quantitative Methods

189.3 Recommendations for Effective Project Evaluation and Selection

A Final Note

Hans J. Thamhain
Bentley College

For most organizations, resources are limited. The ability to select and fund the best projects with the highest probability of success is crucial to an organization's ability to survive and prosper in today's highly competitive environment. Project selections, necessary in virtually every business area, cover activities ranging from product developments to organizational improvements, customer contracts, R&D activities, and bid proposals. Evaluation and selection methods support two principal types of decisions: 1. Judging the chances of success for one proposed project 2. Choosing the best project among available alternatives Although most decision processes evaluate projects in terms of cost, time, risks and benefits, such as shown in Table 189.1, it is often extremely difficult, if not impossible, to define a meaningful aggregate measure for ranking projects regarding business success, technical risks, or profit. Managers can use traditional, purely rational selection processes toward "right," "successful," and "best" only for a limited number of business situations. Many of today's complex project evaluations require the integration of both analytical and judgmental techniques to be meaningful. Table 189.1 Typical Criteria for Project Evaluation and Selection
The criteria relevant to the evaluation and selection of a particular project depend on the specific project type and business situation such as project development, custom project, process development, industry and market. Typically, evaluation procedures include the following criteria: Development cost Development time Technical complexity

1998 by CRC PRESS LLC

 Risk Return on investment Cost benefit Product life cycle Sales volume Market share Project business follow-on Organizational readiness and strength Consistency with business plan Resource availability Cash flow, revenue, and profit Impact on other business activities Each criterion is based on a complex set of parameters and variables.

Although the literature offers a great variety of project selection procedures, each organization has its own special methods. Approaches fall into one of three principal classes: 1. Primarily quantitative and rational approaches 2. Primarily qualitative and intuitive approaches 3. Mixed approaches, combining both quantitative and qualitative methods

189.1 Quantitative Approaches to Project Evaluation and Selection

Quantitative approaches are often favored to support project evaluation and selections if the decisions require economic justification. Supported by numeric measures for simple and effective comparison, ranking, and selection, they help to establish quantifiable norms and standards and lead to repeatable processes. However, the ultimate usefulness of these methods depends on the assumption that the decision parameterssuch as cash flow, risks, and the underlying economic, social, political, and market factorscan actually be quantified. Typically, these quantitative techniques are effective decision support tools if meaningful estimates of capital expenditures and future revenues can be obtained and converted into net present values for comparison. Table 189.2 shows the cash flow of four project options to be used for illustrating the quantitative methods described in this chapter. Table 189.2 Cash Flow of Four Project Options or Proposals*
End of Year 0 1 2 3 4 5 Do-Nothing Option P1 0 0 0 0 0 0 Project Option P2 1000 200 200 200 200 200 Project Option P3 2000 1500 1000 800 900 1200 Project Option P4 5000 1000 1500 2000 3000 4000

1998 by CRC PRESS LLC

Net cash flow NPV|N = 5 NPV|N = ROI|N = 5 CB = ROINPV jn = 5 NPBP ji = 0 NNPV ji

0 0 0 0 0 0 0

0 242 +1000 20% 67% 5 7.3

+3400 +2153 +9904 54% 108% 1.5 5

+7 500 +3 192 +28 030 46% 164% 3.3 3.8

*Assuming an MARR of i = 10% Note: The first line of negative numbers represents the initial investment at the beginning of the life cycle.

Net Present Value (NPV) Comparison

This method uses discounted cash flow as the basis for comparing the relative merit of alternative project opportunities. It assumes that all investment costs and revenues are known and that economic analysis is a valid singular basis for project selection. We can determine the net present value (NPV) of a single revenue, stream of revenues, and/or costs expected in the future. Present worth of a single revenue or cost (often called annuity, A) occurring at the end of period n and subject to an effective interest rate i (sometimes referred to as discount rate or minimum attractive rate of return, MARR) can be calculated as:

P W (A j i; n) = A

1 = P Wn (1 + i)n

Net present value of a series of revenues or costs, An, over N periods of time is as follows:

NPV(An j i; N ) =

N X n=1

X 1 An = P Wn (1 + i)n n=1

Table 189.2 applies these formulas to four project alternatives, showing the most favorable 5-year net present value of $3192 for project option P4. (There are three special cases of net present value: (1) for a uniform series of revenues or costs over N periods, NPV(Anj i, N) = A[(1+i)N 1]/i(1+i)N; (2) for an annuity or interest rate i approaching zero, NPV = A N; and (3) for the revenue or cost series to continue forever, NPV = A/i.)

Return-on-Investment Comparison
Perhaps one of the most popular measures for project evaluation is the return on investment (ROI):

ROI =

Revenue (R) Cost (C) Investment (I)

1998 by CRC PRESS LLC

It calculates the ratio of net revenue over investment. In its simplest form it entails the revenue on a year-by-year basis relative to the initial investment (for example, project option 2 in Table 189.2 would produce a 20% ROI). Although this is a popular measure, it does not permit a comparative evaluation of alternative projects with fluctuating costs and revenues. In a more sophisticated way we can calculate the average ROI per year # " N X An =N ROI(An ; In j N ) = In
n=1

and compare it to the minimum attractive rate of return. All three project options, P2, P3 and P4, compare favorably to the MARR of 10%, with project P3 yielding the highest average return on investment of 54%. Or we can calculate the net present value of the total ROI over the project lifecycle, also known as cost-benefit (CB). This is an effective measure of comparison, especially for fluctuating cash flows. (Table 189.2 shows project 3 with the highest 5-year ROINPV of 108%.) " N # " N # . X X ROINPV (An ; In j i; N ) = NPV(An j i; N ) NPV(In j i; N )
n=1 n=1

Pay-Back Period (PBP) Comparison

Another popular figure of merit for comparing project alternatives is the payback period (PBP). It indicates the time period of net revenues required to return the capital investment made on the project. For simplicity, undiscounted cash flows are often used to calculate a quick figure for comparison, which is quite meaningful if we deal with an initial investment and a steady stream of net revenue. However, for fluctuating revenue and/or cost streams, the net present value must be calculated for each period individually and cumulatively added up to the "break-even point" in time, NPBP, when the net present value of revenue equals the investment.
N X n=1

NPV(An j i)

N X n=1

NPV(In j i)

Pacifico and Sobelman Project Ratings

The previously discussed methods of evaluating projects rely heavily on the assumption that technical and commercial success is ensured and all costs and revenues are predicable. Because of these limitations, many companies have developed their own special procedures for comparing project alternatives. Examples are the project rating factor (PR), developed by Carl Pacifico for

1998 by CRC PRESS LLC

assessing chemical products, and the project value factor (z), developed by Sidney Sobelman for new product selections:

PR =

pT pC R TC

z = (P TLC ) (C TD )

Pacifico's formula is in essence an ROI calculation adjusted for risk. It includes probability of technical success [.1 < pT < 1.0], probability of commercial success [.1 < pC < 1.0], total net revenue over project life cycle [R], and total capital investment for product development, manufacturing setup, marketing, and related overheads [TC]. The Sobelman formula is a modified cost-benefit measure. It takes into account both the development time and the commercial life cycle of the product. It includes average profit per year [P], estimated product life cycle [TLC], average development cost per year [C], and years of development [TD].

Limitations of Quantitative Methods

Although quantitative methods of project evaluation have the benefit of producing relatively quickly a measure of merit for simple comparison and ranking, they also have many limitations, as summarized in Table 189.3. Table 189.3 Comparison of Quantitative and Qualitative Approaches to Project Evaluation
Quantitative Methods Benefits: Simple comparison, ranking, selection Repeatable process Encourages data gathering and measurability Benchmarking opportunities Programmable Input to sensitivity analysis and simulation Qualitative Methods Benefits: Search for meaningful evaluation metrics Broad-based organizational involvement Understanding of problems, benefits, opportunities Problem solving as part of selection process Broad knowledge base Multiple solutions and alternatives Multifunctional involvement leads to buy-in Risk sharing Limitations: Complex, time-consuming process Biases via power and politics Difficult to proceduralize or repeat Conflict- and energy-intensive Do not fit conventional decision processes Intuition and emotion dominates over facts Justify wants over needs Lead to more fact finding than decision making

Limitations: Many success factors are nonquantifiable Probabilities and weights change True measures do not exist Analysis and conclusions are often misleading Methods mask unique problems and opportunities Stifle innovative decision making Lack people involvement, buy-in, commitment Do not deal well with multifunctional issuesand dynamic situations Pressure to act prematurely

1998 by CRC PRESS LLC

189.2 Qualitative Approaches to Project Evaluation and Selection

Especially for project evaluations involving complex sets of business criteria, the narrowly focused quantitative methods must often be supplemented by broad-scanning; intuitive processes; and collective, multifunctional decision making, such as Delphi, nominal group technology, brainstorming, focus groups, sensitivity analysis, and benchmarking. Each of these techniques can either be used by itself to determine the best, most successful, or most valuable option, or be integrated into an analytical framework for collective multifunctional decision making, which is discussed in the next section.

Collective Multifunctional Evaluations

Collective multifunctional evaluations rely on subject experts from various functional areas for collectively defining and evaluating broad project success criteria, employing both quantitative and qualitative methods. The first step is to define the specific organizational areas critical to project success and to assign expert evaluators. For a typical product development project, these organizations may include R&D, engineering, testing, manufacturing, marketing, product assurance, and customer services. These function experts should be given the time necessary for the evaluation. They also should have the commitment from senior management for full organizational support. Ideally, these evaluators should have the responsibility for ultimate project implementation, should the project be selected. The next step is for the evaluation team to define the factors that appear critical to the ultimate success of the projects under evaluation and arrange them into a concise list that includes both quantitative and qualitative factors. A mutually acceptable scale must be worked out for scoring the evaluation criteria. Studies of collective multifunctional assessment practices show that simplicity of scales is crucial to a workable team solution. Three types of scale have produced most favorable results in field studies: (1) 10-point scale, ranging from +5 = most favorable to -5 = most unfavorable; (2) 3-point scale, +1 = favorable, 0 = neutral or can't judge, -1 = unfavorable; and (3) 5-point scale, A = highly favorable, B = favorable, C = marginally favorable, D = most likely unfavorable, F = definitely unfavorable. Weighing of criteria is not recommended for most applications, since it complicates and often distorts the collective evaluation. Evaluators score first individually all of the factors that they feel qualified to make an expert judgment on. Collective discussions follow. Initial discussions of project alternatives, their markets, business opportunities, and technologies involved are usually beneficial but not necessary for the first round of the evaluation process. The objective of this first round of expert judgments is to get calibrated on the opportunities and challenges presented. Further, each evaluator has the opportunity to recommend (1) actions needed for better assessment of project, (2) additional data needed, and (3) suggestions that would enhance project success and the evaluation score. Before meeting at the next group session, agreed-on action items and activities for improving the decision process should be completed. With each iteration the function-expert meetings are enhanced with more refined project data. Typically, between three and five iterations are required before a project

1998 by CRC PRESS LLC

selection can be finalized.

189.3 Recommendations for Effective Project Evaluation and Selection

Effective evaluation and selection of project opportunities is critical to overall project success. With increasing complexities and dynamics of the business environment, most situations are too complex to use simple economic models as the sole basis for decision making. To be effective, project evaluation procedures should include a broad spectrum of variables for defining the project value to the organization. Structure, discipline, and manageability can be designed into the selection process by grouping the evaluation variables into four categories: (1) consistency and strength of the project with the business mission, strategy, and plan; (2) multifunctional ability to produce the project results, including technical, cost, and time factors; (3) success in the customer environment; and (4) economics, including profitability. Modern phase management and stage-gate processes provide managers with the tools for organizing and conducting project evaluations effectively. Table 189.4 summarizes suggestions that may help managers to effectively evaluate projects for successful implementation. Table 189.4 Suggestions for Effective Project Evaluation and Selection 1. Seek out relevant information. Meaningful project evaluations require relevant quality information. The four categories of variables can provide a framework for establishing the proper metrics and detailed data gathering. 2. Take top-down look; detail comes later. Detail is less important than information relevancy and evaluator expertise. Don't get hung up on lack of data during the early phases of the project evaluation. Evaluation processes should iterate. It does not make sense to spend a lot of time and resources gathering perfect data to justify a "no-go" decision. 3. Select and match the right people. Whether the project evaluation consists of a simple economic analysis or a complex multifunctional assessment, competent people from those functions critical to the overall success of the project(s) should be involved. 4. Success criteria must be defined. Deciding on a single project or choosing among alternatives, evaluation criteria must be defined. They can be quantitative, such as ROI, or qualitative, such as the probability of winning a contract. In either case, these evaluation criteria should cover the true spectrum of factors affecting success and failure of the project(s). Only functional experts, discussed in point 3, are qualified to identify these success criteria. Often, people from outside the company, such as vendors, subcontractors, or customers, must be included in this expert group. 5. Strictly quantitative criteria can be misleading. Be aware of evaluation procedures based only on quantitative criteria (ROI, cost, market share, MARR, etc). The input data used to calculate these criteria are likely based on rough estimates and are often unreliable.

1998 by CRC PRESS LLC

Evaluations based on predominately quantitative criteria should at least be augmented with some expert judgment as a "sanity check." 6. Condense criteria list. Combine evaluation criteria, especially among the judgmental categories, to keep the list manageable. As a goal, try to stay within 12 criteria. 7. Communicate. Facilitate communications among evaluators and functional support groups. Define the process for organizing the team and conducting the evaluation and selection process. 8. Ensure cross-functional cooperation. People on the evaluation team must share a strategic vision across organizational lines. They also must sense the desire of their host organizations to support the project if selected for implantation. The purpose, goals, and objectives of the project should be clear, along with the relationship to the business mission. 9. Don't lose the big picture. As discussions go into detail during the evaluation, the team should maintain a broad perspective. Two global judgment factors can help to focus on the big picture of project success: (1) overall benefit-to-cost perception and (2) overall risk-of-failure perception. These factors can be recorded on a ten-point scale: 5 to +5. This also leads to an effective two-dimensional graphic display of competing project proposals. 10. Do your homework between iterations. As project evaluations are most likely conducted progressively, action items for more information, clarification, and further analysis surface. These action items should be properly assigned and followed up, thereby enhancing the quality of the evaluation with each consecutive iteration. 11. Stimulate innovation. Senior management should foster an innovative ambience for the evaluation team. Evaluating complex project situations for potential success or failure involves intricate sets of variables, linked among organization, technology, and business environment. It also involves dealing with risks and uncertainty. Innovative approaches are required to evaluate the true potential of success for these projects. Risk sharing by senior management, recognition, visibility, and a favorable image in terms of high priority, interesting work, and importance of the project to the organization have been found strong drivers toward attracting and holding quality people to the evaluation team and gaining their active and innovative participation in the process. 12. Manage and lead. The evaluation team should be chaired by someone who has the trust, respect, and leadership credibility with the team members. Further, management can positively influence the work environment and the process by providing some procedural guidelines, charters, visibility, resources, and active support to the project evaluation team.

A Final Note
Effective project evaluation and selection requires a broad-scanning process that can deal with the risks, uncertainties, ambiguities, and imperfections of data available at the beginning of a project cycle. It also requires managerial leadership and skills in planning, organizing, and

1998 by CRC PRESS LLC

communicating. Above all, evaluation team leaders must be social architects in unifying the multifunctional process and its people. They must share risks and foster an environment that is professionally stimulating and strongly linked with the support organizations eventually needed for project implementation. This is an environment that is conducive to cross-functional communication, cooperation, and integration of the intricate variables needed for effective project evaluation and selection.

Defining Terms
Cross-functional: Actions that span organizational boundaries. Minimum attractive rate of return (MARR): The annual net revenue produced on average by projects in an organization as a percentage of their investments. Sometimes MARR is calculated as company earnings over assets. Net worth: Discounted present value of a future revenue or cost. Phase management: Projects are broken into natural implementation phases, such as development, production, and marketing, as a basis for project planning, integration, and control. Phase management also provides the framework for concurrent engineering and stage-gate processes. Project success: A comprehensive measure, defined in both quantitative and qualitative terms, that includes economic, market, and strategic objectives. Stage-gate process: Framework for executing projects within predefined stages (see also phase management) with measurable deliverables (gate) at the end of each stage. The gates provide the review metrics for ensuring successful transition and integration of the project into the next stage. Weighing of criteria: A multiplier associated with specific evaluation criteria.

References
Brenner, M. 1994. Practical R&D project prioritization. Res. Technol. Manage. 37(5):3842. Bulick, W. J. 1993. Project evaluation procedures. Cost Eng. 35(10):2732. Menke, M. M. 1994. Improving R&D decisions and execution. Res. Technol. Manage. 37(5):2532. Obradovitch, M. M. and Stephanou, S. E. 1990. Project Management: Risk and Productivity. Daniel Spencer, Bend, OR. Remer, D. S., Stokdyk, S. B., and Van Driel, M. 1993. Survey of project evaluation techniques currently used in industry. Int. J. Prod. Econ. 32(1):103115. Schmidt, R. L. 1993. A model for R&D project selection. IEEE Trans. EM. 40(4):403410. Shtub, A., Bard, J. F., and Globerson, S. 1994. Project Management: Engineering, Technology, and Implementation. Prentice Hall, Englewood Cliffs, NJ. Skelton, M. T. and Thamhain, H. J. 1993. Concurrent project management: a tool for technology transfer. Proj. Manage. J. 26(4):4148. Ward, T. J. 1994. Which product is BE$T? Chemical Eng. 101(1):102107.

1998 by CRC PRESS LLC

Further Information
The following journals are good sources of further information: Engineering Management Journal (ASEM), Engineering Management Review (IEEE), Industrial Management (IIE), Journal of Engineering and Technology Management, Project Management Journal (PMI), and Transactions on Engineering Management (IEEE). The following professional societies present annual conferences and specialty publications that include discussions on project evaluation and selection: American Society for Engineering Management (ASEM), Rolla, MO 65401, (314) 341-2101; Institute of Electrical and Electronic Engineers (IEEE), East 47 St., New York, NY 10017-2394; and Project Management Institute (PMI), Upper Darby, PA 19082, (610)734-3330.

1998 by CRC PRESS LLC