PROPOSAL OF A MULTICRITERIA PERFORMANCE

EVALUATION
FOR OUTSOURCED PROJECT PROVIDERS

Edilson Giffhorn1, Leonardo Ensslin1, Sandra Rolim Ensslin1, William Barbosa

Vianna1
1
Santa Catarina Federal University – Universidade Federal de Santa Catarina (UFSC)

Abstract

This article depicts a study on the construction of a personalized performance

evaluation model for a Project Manager of a telecommunications system
operator in Brazil, to serve as support tool for the decision to approve
outsourced service providers. Considering that this context requires broader
understanding, the adopted intervention tool was Multicriteria Decision Aid -
Constructivist. As a result, the telecommunications operator and Project
Manager were provided with a management tool that allows dissemination of
outsourced providers regarding relevant factors and how these factors can
improve their performance, according to values and preferences of the decision-
maker of the telecommunications operator.

KEYWORDS: Multicriteria Decision Aid - Constructivist, Management, Outsourcing,

Performance evaluation.

1. Introduction

This article, developed as a Case Study, describes the construction of a multicriteria

performance evaluation model applicable to outsourced telecommunications providers of
Brazil. Construction of this model provides project managers with a tool that supports their
decision to fully approve outsourced companies, and a performance enhancement
management tool for companies with previous approval.

In the last two decades, increased demand for the evolution of infrastructure in Brazil
has resulted in a rapid nationwide expansion of telecommunications services. Consequent
market pressure has forced companies to restructure in order to remain competitive, and bring
new services, namely in mobile communication, the corporate market and data
communication access. To survive this new context, companies have adopted strategies for
cost reduction, higher productivity, enhanced quality, service diversification and higher
shareholder return. One of the means adopted to make these objectives operable has been
Outsourcing, resulting in a network of service providers for telecommunications operators.
This context also comprises the organization in which this research was conducted. In
this company, outsourced providers needed approval of the corresponding branch project
manager in order to act in data communication systems expansion projects. The decision,

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

however, was largely based on intuitive criteria. Research showed that the absence of an
evaluation model that clearly defined adopted criteria for approval of outsourced providers
exposed the operator to performance problems that compromised quality of services, caused
additional costs and damaged the public image of the operator.
The research question was: which are the key performance indicators to be considered
in the approval process of outsourced providers in order to meet the specific needs of a
telecommunications project manager?
To answer this question, the study sought to build a personalized performance
evaluation model for the decision-maker (project manager) of the telecommunications
operator in question. Consequently, specific objectives were: (i) contextualize the decision-
making environment, identify actors involved and create a label to define the problem; (ii)
identify essential evaluation elements from the decision-maker’s viewpoint and determine the
cause-effect relation between these element; (iii) evaluate outsourced providers; (iv) provide
understanding of the decisional context of the decision-maker to allow acknowledgement of
the impact that potential actions may have on performance of evaluation criteria.
Exploratory research presented herein describes a Case Study conducted in the
department of data communication of a branch of a large telecommunications operator in the
South of Brazil. The adopted intervention tool was Multicriteria Decision Aid - Constructivist
(MCDA-C) approach, due to the differences in relation to other MCDA methods to
structuring contexts.
Primary data was collected from the operator project manager through non-structured
interviews and in-company documents. Research methodology was quali-quantitative.
Qualitative was adopted to structure the problem, identify criteria, build ordinal scales and
generate enhancement recommendations. Quantitative was adopted for the transfer of ordinal
scales to cardinal scales and consequent integration.
This article is divided in five sections: section 1 contains this introduction; section 2
comprises a short contextualization of Outsourcing and Performance Evaluation; section 3
contains the theoretical reference of MCDA-C; section 4 shows the Case Study and section 5
presents Final Considerations.

2. Performance Evaluation of Outsourced Providers

From the moment in which competitiveness started ignoring frontiers, enabling the
opening of protected markets to global competition, companies have been forced to rethink
operation methods to assure survival in a new scenario. Adopted strategies to confront this
challenge of increasing dynamicity include Outsourcing.
The concept of outsourcing has thus evolved from simple transfer of activities of no
financial interest to a source of specialization and differential competitiveness. This evolution
displaced the focus of cost reduction and quality improvement to include performance
enhancement and added value to customers.
In Brazil, outsourcing has gained force since the start of the 1990s, with the end of
market reserve laws, start of public service privatizations, increase of fixed costs, tax burden
and number of inputs needed for support activities. In telecommunications, outsourcing
became a viable option with the privatization of state companies in 1998. Since then, the use

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

of outsourced providers for maintenance and system extension projects was considered a
distinctive asset in the new highly competitive national telecommunications scenario.
Once telecommunications operators became dependent on performance of outsourced
providers, the need for a personalized organizational process, which formalized company
approval procedures and provided a strategic objectives alignment method of the involved
companies, became vital for the success of this commercial relationship.
Personalized creation of a performance evaluation model is a research specification.
Consequently, the proposed intervention instrument is Multicriteria Decision Aid -
Constructivist (MCDA-C) approach as a form of acting in the Case Study context, thus
providing the means to perfect alignment of operational actions with strategic objectives.

3. Multicriteria Decision Aid - Constructivist

Decision Aid arises from the presumption that decisions are complex and
personalized, and that stakeholders do not have the understanding that allows prevision of
consequences of their decision in their system of values (Roy and Vanderpootem, 1996).

The Multicriteria Decision Aid (MCDA) method was consolidated as a management

tool in the 1980s, with the works of Roy (1996) and Landry (1995), who defined objectivity
limits for decision aid processes; Skinner (1986) and Keeney (1992), who acknowledged that
attributes are specific to the decision-maker in each context; and Bana and Costa (1993), who
explained the convictions of MCDA.
MCDA-C approach emerged as a ramification of traditional multicriteria
methodologies (AHP, MAVT, MAUT, MCDA, SMART) used to support decision-makers in
complex (unexplained multiple variables), conflicting (multiple actors with different interests)
and uncertain contexts (restricted knowledge of necessary information for a decision).
The essential distinction between MCDA-C and other traditional MCDA methods is
that MCDA restricts decision aid to the formulation stage and evaluation for selection, within
a set of preset alternatives, of which is the best option or which provides an optimal solution
(Keeney, 1992; Roy & Bouyssou, 1993; Roy, 1996; Goodwin & Wright, 1998). MCDA-C
approach, on the other hand, acknowledges limits of objectivity and makes them operational
by means of open interviews, brainstorming, means-end relation maps and optimization
models, amongst other instruments. According to Roy (1994, 1996, 2005), MCDA
researchers can be classified in two groups: those who take a rationalist standpoint and those
who adopt constructivism as logic of investigation. By adopting constructivism, MCDA-C
allows the decision-maker to understand the decisional context, enabling him or her to
comprehend the consequences of decision in aspects considered important. In these cases, the
decision-maker needs:
• Support to explicit and measure his/her values and preferences, as opposed to generic
values and preferences;
• Comprehend and visualize the consequences of his/her decisions within objectives;
• Determine reference performance in each objective according to perception;
• Understand contribution of each objective within strategic objectives;
• Broaden knowledge to identify enhancement opportunities.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 The knowledge development process of the decision-maker in MCDA-C comprises
three stages: Structuring, Evaluation and Recommendations. The Structuring stage identifies,
organizes and measures, in ordinal values, the concerns that the decision-maker considers
necessary and sufficient to evaluate the context. Evaluation is used to improve understanding
for construction of cardinal scales and replacement rates that represent local and global
preferences. Finally, Recommendations broadens understanding by seeking to comprehend
consequences of possible decisions in presumably relevant criteria, and within the context as a
whole.

4. Case Study

This item discusses the research Case Study in three sections, in accordance with
stages of the MCDA-C model.

4.1 Structuring

The Case Study was conducted in a telecommunications operator with activity in the
South of Brazil. In this company, expansion projects of data communication systems are
performed by outsourced providers. The company has around six thousand collaborators, and
more than 10 million customers. Data communication expansion projects are conducted in the
parent company, which is also responsible for drafting contracts for outsourced project
providers. Projects approved in the parent company are managed in each branch by the
corresponding project managers, who directly interact with outsourced company technicians
responsible for execution of works. In order to operate, outsourced companies need approval
of the project manager of each branch, who issues his/her decision and sends it to the parent
company. Subsequently, the outsourced provider starts activities with temporary approval
until the final decision of the recommended manager to continue, or not, provision of
services.
The evaluation process of outsourced companies under the responsibility of the project
manager, however, was essentially based on intuition. The evaluation procedure used to
decide approval or non-approval of companies did not observe requirements considered
necessary. This procedure showed fragility and resulted in unfounded and unjustified
company approval, lacking decision history or feedback of outsourced companies that could
enable performance enhancement.
Consequently, the actors involved in this context were identified, which are: (i)
decision-maker: project manager; (ii) stakeholders: telecommunications operator auditors,
network directories, suppliers, fault management, operations and maintenance management;
Anatel (Agência Nacional de Telecomunicações [National Telecommunications Agency]);
(iii) the agents: customers, society and outsourced technicians; (iv) facilitators: authors of the
article.
An interactive process between facilitators and the decision-maker was subsequently
adopted to create a Label that closely represented decision-maker concerns regarding the
problem, which in this case was defined as: creation of a structured process that evaluates
performance of outsourced providers.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Two open interviews were conducted with decision-maker, who was asked to talk
about the problem. Their statements were recorded, transcribed and analysed to identify the
elements the decision-maker considered essential for the evaluation model being created,
called Primary Assessment Elements (PAE). One of the 60 identified PAEs was: Information
of previously occupied space.
Each PAE requires identification of preferred direction, psychological opposite and
minimum level of acceptance of the subjacent objective. This method of presenting PAEs is
called Concept (Eden & Ackermann, 1992). The Concept corresponding to the PAE specified
above is: Do not use space reserved for other equipment without consultation... payment
blocked due to installation in incorrect location.
In the Concept, reticence (...) must be read as “preferably” or “instead of” and
corresponds to the psychological opposite.

Concepts are grouped according to respective concerns, which generate Areas of

Concern, and each receives a denomination according to the main concern of the decision-
maker during grouping. In sequence, MCDA-C leads to broadening of knowledge using
identification of hierarchy relations and influence among concepts. Means-End Relations
Maps are used for this purpose (Bana and Costa et al., 1999; Ensslin, Dutra & Ensslin, 2000),
which are created by asking the decision-maker to consider, for each concept: “How can the
end concept be obtained?” and “Why is the means concept important?”
Concept 25 is considered for illustration purposes – “Assure execution of works
complies with designed works... can cause operational faults to customer”, which is shown in
Figure 1.
Facilitators asked the decision-maker to specify how this Concept could be obtained.
The decision-maker considered that Concept 25 could be obtained through Concept 17 –
“Install in location designated in project... have space conflicts with other departments”,
Concept 7 – “Identify installations... receive Anatel fines”, and a new Concept that emerged
during the relations determination process, “Assure correct installation of equipment.....
fixture, grounding and positions does not comply with project”. Attention was then focused
on ends, namely facilitators asked the decision-maker why Concept 25 was important. The
decision-maker considered that Concept 25 was important as it dealt with one of the forms to
obtain Concept 19 – “Meet internal normative standards... delay payments due to pending
issues”.
This process was repeated for each concept until all cause-effect ratios and Means-
Ends Maps were obtained. In order to facilitate analysis and understanding, the Means-End
Map was divided into Clusters, formed by grouping branches with argumentation that reflects
one same decision-maker concern. Figure 1 partially shows the Means-Ends Map for the
Compliance with Standards Cluster.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Evaluate Outsource Company
Performance

Meet installation
standards....have internal
problems and with Anatel

19. Meet internal normative 24. Meet Anatel

Compliance standards standards...
with Standards ... Delay payments due to Receive Anatel
Cluster Internal paterns pending issues fines
Anatel
Subcluster Subcluster
4. Consider future expansions 25. Assure execution of works
during cable installation... complies with designed works
Extend chutes and rearrange ... Can cause operational faults
cables to customer

Assure correct installation of 17. Install in location

7. Identify
equipment..... fixture, designated in project...
installations...
grounding and positions does Have space conflicts with
Receive Anatel fines
not comply with project other departments
Instalation Identification Location
Subcluster Subcluster Subcluste
40. r
38. Efficient 43. Do not use
13. Assure Disclose 42. Speed up
14. Assure 39. Enough auditing... area reserved for
Assure project grounding is Anatel installation to
equipment auditor for Start other equipment
delimitation of adequate... standards assure use of
is well auditing... installation without prior
location at Rework due for delimited
fixed... Delay or non without authorization...
plant in not to grounding outsource space... Lose
Rework to verification of previous Payments blocked
occupied... in the providers... space reserved
correct identification auditing due to installation
Rework for incorrect Allow them due to delay at
fixture (labeling) in incorrect
relocation location to define start of
location
standards installation

Figure 1 – Partial of the Means-Ends Map for Compliance with standards Cluster.

The following step of the MCDA-C approach consists of converting the structure of
Means-End Maps to a Hierarchy Structure of Value. Each cluster should be tested to meet the
following properties: essential, controllable, complete, measurable, operational, isolable, non-
redundant, concise and comprehensible (Keeney, 1992; Ensslin, Montibeller & Noronha,
2001; Roy, 2005). Clusters that meet these properties receive the denomination Fundamental
Point of View (FPV).

FPVs, however, are too broad to be measured. For this reason, returns to Means-Ends
Maps in order to analyze the sub clusters. Their transformation process is equivalent to that of
FPVs. This subcriterion is denominated Elementary Points of View (EPVs). The breakdown
process continues until EPV is reached which represents a context priority and thus can be
measured in an objective and unambiguous manner. Figure 2 shows the Arborescence for
first-level EPVs of the created model.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 2 - Points of View Arborescence of the model.

Once the Hierarchy Structure of Value is built, the next step is the creation of ordinal
scales to measure Points of View. These ordinal scales are called, Descriptors.
Ordinal scales are built using an interactive process that also counts on active
participation of the decision-maker. At this moment, the decision-maker creates a scale that
best represents that which he/she judges relevant, and identifies the Reference Levels (Good
Level – above which performance is excellent; Neutral Level – below which performance is
endanger). Between these two points is considered market performance (Roy, 2005).
Figure 3 partially shows the Hierarchy Structure of Value for FPV Patterns Serving,
with corresponding EPVs and Descriptors.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 3 – Partial of Hierarchy Structure of Value for FPV Patterns Serving.

In order to continue development of context understanding, information that allows

transformation of ordinal into cardinal scales should be incorporated, which occurs in the
following stage of MCDA-C approach, the Evaluation.

4.2 Evaluation

Ordinal scales created for descriptors often contain representative numeric symbols,
which are not numbers of the ℜ set of real numbers, (Barzilai, 2001; Azevedo, 2001).
Therefore, these scales should not be used for functions that involve numeric operations.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

MCDA-C acknowledges the differences between ordinal and cardinal scales and, in order to
obtain cardinality, asks the decision-maker to make semantic judgements that allow detection
of the attractiveness difference between levels of each scale. Transformation of scales can be
performed using different methods, but the selected method in this study was Macbeth
(Measuring Attractiveness by a Categorical Based Evaluation Technique) (Bana e Costa, De
Corte & Vansnick, 2005), due to its theoretical basis, representativity and practicality. It should
be emphasized that Macbeth is a method to transform ordinal scales into cardinal scales from
judgements on the difference of attractiveness between two alternatives, and it is not a
decision aid method.

Information on attractiveness differences supplied by the decision-maker is inserted in

the M-Macbeth software to obtain the corresponding cardinal scales that meet the preference
judgements of the decision-maker. These scales are called Value Functions.
Figure 4 shows the transformation process of a descriptor (ordinal scale) into a Value
Function (cardinal scale): Descriptor D17 – Prior Auditing; Judgement Matrix; the anchored
scales supplied by M-Macbeth software and the Value Function, numerical and graphic.

Figure 4 – Transformation of Prior Auditing Descriptor into Value Function using Macbeth
Methodology.

Figure 5 portrays the partial set of cardinal scales for FPV Pattern serving.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 5 – Cardinal scales of FPV Pattern serving.

To meet the conditions for evaluation that allows performance comparison of different
outsourced companies, local values of each criterion are added to a global evaluation value,
thus allowing performance measurements of each outsourced company. Integration is possible
using Substitution Rates. In order to prevent the decision-maker from expressing numeric
judgement values, the Macbeth comparison method was adopted to provide semantic values.
Firstly, the desired hierarchical structure is identified. Figure 6 highlights the
hierarchical structure for better illustration of the process, which is the determination of rates
for EPVs – Project Conformity, Document Markings, Timeline for Start of Installation and
Prior Auditing.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 6 – Hierarchy Structure of Value with emphasis on EPVs Project Conformity,
Document Markings, Timeline for Start of Installation and Prior Auditing.

Potential actions that represent the contribution of the transfer from Neutral Level to
Good Level are created in each desired criterion substitution rate, along with reference actions
with Neutral performance in all points of view, as specified in Figure 7. In sequence, the list
of alternatives is ordered according to preferences of the decision-maker. Listed alternatives
and semantic judgements of the attractiveness difference after switching one action for
another are inserted in M-Macbeth software, which provides the Substitution Rates.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 7 – Potential alternatives to determine Substitution Rates with indication of
corresponding Good and Neutral Levels.

The same procedure is adopted with the other hierarchical structures. Figure 8 shows
the Hierarchy Structure of Value with Substitution Rates of the created model.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Routes

Figure 8 – Hierarchy Structure of Value with Substitution Rates.

In order to globally evaluate the impact of an action, local evaluations of each action
should be added, providing a weighted sum with punctuation obtained by the action in each
criteria multiplied by the corresponding criteria weight. This results in a global equation of
the Case Study model, which is:

In equation (1), constants are obtained by the product of rates in the ascending
Hierarchy Structure of Value. The generic integration equation form is:

V FPV k (a ) = ∑ w i ,k • v i ,k (a )
nk

i =1 (2)

In which:
V FPVk (a ) : global value of action a of FPVk, k = 1,… m;
vi,k(a): partial value of action a of criterion i, i = 1,...n, of FPVk, k = 1,… m;
a: level of impact of action a;
wi,k : substitution rate of criterion i, i = 1,... n, of FPVk, k = 1,… m;
nk: number of criteria of FPVk, k = 1,… m;
m: number of FPVs to the model.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 From this moment, the model can be used for decision aid in the outsourced provider
management process.
The decision-maker used the model to enhance alignment of objectives with
outsourced providers that have already been approved and as a support tool in the final
decision issuance process for companies under approval. Subsequently, the decision-maker
selected three real service providers. Two of these companies (company 1 and 2) had been
approved two years ago, and the third (company 3) was in the period of acceptance and had
been providing services for three months. The decision-maker used the global model as
support for each company and pinpointed the performance status quo in each criterion, thus
allowing construction of an impact profile of each criterion shown in Figure 9. Companies
used in the study are real, although punctuation values were altered to prevent similarity to
real values, assuring confidentiality of company information.

Figure 9 – Impact Profile with status quo of the three evaluated outsourced providers.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Impact profile analysis shows no significant differences between performance of the
evaluated companies. However, this representation allows verification of criterion strong
points and weak points of each evaluated company, enabling clarification of which criterion
provide main improvement opportunities and transparent and justified feedback to companies.
Armed with an understanding of where action is convenient, the project manager can
now identify enhancement actions and evaluate their impact in the range of strategic
objectives. This process is executed in the Recommendations stage.

4.3 Recommendations

The function of the Recommendations stage is to support the decision-maker in the

identification of ways to performance improvement of that which is being evaluated, and
provide an understanding of the consequences of these actions in case they are implemented.
This stage does not have a prescriptive nature, but the character of aid the decision-maker to
build actions and understand their consequences.
Their implementation occurs with the identification of FPVs where performance
enhancement is desired. FPV 6 – Technical Communication was considered to illustrate the
process.
The process comprises visualization of the descriptor and impact level status quo of
the evaluated company. Subsequently, facilitators used brainstorming sessions with involved
actors to seek alternatives for EPV impact to present improvement. An example is the EPV
description – Incorrect Information, as shown in Figure 10.

Figure 10 – Performance alteration of EPV Descriptor – Incorrect Information with

application of Alpha Actions

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Five actions that allow elevation of EPV performance were verified – Incorrect
Information of 10 occurrences to 1 occurrence for company 3. Here, values are also
hypothetical to preserve company information. An equal procedure was adopted with the
remaining EPVs of the FPV. The set of actions, if implemented, would raise FPV
performance – Technical Communication from +46 to a presumed value of +80 for company
3.
In this way, the Recommendations stage provides conditions for the decision-maker to
identify: (i) where it is convenient to act; (ii) a process to generate actions that promote
enhancement; (iii) visualization of consequences of enhancement actions at local level (EPV),
tactic level (FPV) and strategic level (global).
Potentially created actions were grouped in enhancement strategies, with the following
focal points:
• Qualification and training actions: continued training; instructive rather than punitive
auditors; promotion of required qualifications, abilities and skills;
• Communication actions: alignment meetings; improvement of communication
channels; evaluations feedback; receptivity to enhancement suggestions proposed by
outsourced technicians;
• Quality actions: training of inter-company groups for quality enhancement
propositions; monitoring of fault indicators;
• Computer actions: development of integrated project management web system; online
performance evaluation model punctuation system;
• Social interaction actions: Procedures manual; social conduct enhancement;
adaptations for the handicapped.

5. Final Considerations

The objective of this article was to show the construction of a performance evaluation
model of outsourced providers of a Brazilian telecommunications operator. The MCDA-C
approach was adopted as intervention instrument due to the complexity of the context, with
conflicting interests, in which the decision-maker had no clear knowledge of the criteria that
should be included in the performance evaluation model. Data used for model construction
was obtained through non-structured interviews with a project manager of one of the
company branches and contractual documents. Research was exploratory, in the form of Case
Study, with a qualitative and quantitative approach.
The participative decision aid process allowed the decision-maker to identify,
organize, measure and integrate aspects he/she considered necessary and sufficient to evaluate
outsourced company performance. In this way, the project manager had the conditions to: (i)
provide feedback to companies regarding the criteria considered relevant in the evaluation;
(ii) know local punctuation of each criterion of each company; (iii) know global evaluation of
companies; (iv) compare company performances; (v) use an instrument that justified issuance
of final approval, or not, of the company that provided services in the acceptance period; and,
(vi) acquire knowledge of the impact of potential actions in the performance of each
company.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

 Performance evaluation models created with the use of MCDA-C are personalized to
actors, context and moment. Consequently, contextualization and full description of actors
was required, with labelling of the problem according to decision-maker perception, as stated
in section 4.1. As a result, specific objective 1 was reached, which was contextualization of
the decision-making environment, identification of actors involved and creation of a label to
define the problem.
Section 4.1 also shows the construction process of Means-End Maps and of
Arborescence of Fundamental Points of View which specify key criteria for the evaluation of
outsourced provider performance, namely: 1- Patterns Serving; 2- Professional Qualification;
3- Tests; 4- Priorities; 5- Routes; 6- Technique; and 7- Organizational. This led to the
attainment of specific objective 2, which is identification of essential evaluation elements
from the decision-maker’s viewpoint and determine the cause-effect relation between these
elements.
This resulted in the obtainment of the answer to the research question: “which are the
key performance indicators to be considered in the approval process of outsourced providers
in order to meet the specific needs of a telecommunications project manager?”
Outsourced companies could be comparatively evaluated, as shown in the Impact
Profile of Figure 9, attaining specific objective 3 of evaluate outsourced providers.
Lastly, the creation process of potential enhancement actions is shown in section 4.3,
Recommendations, allowing attainment of specific objective 4, provide understanding of the
decisional context of the decision-maker to allow acknowledgement of the impact that
potential actions may have on performance of evaluation criteria.
The created model was legitimized by the project manager in each stage of its
construction. Use of this model provided the decision-maker and the telecommunications
operator with an instrument that allows continuous improvement by means of creative and
innovative performance enhancement actions.

Acknowledgment

The authors would like to thank to Cipriana Leme da Silva who made the translation
of this paper.

References

Azevedo, J. (2001). “Aplicação da metodologia multicritério de apoio à decisão na seleção de centros

de usinagem para uma central de usinagem”. Santa Catarina Federal University, Thesis (Master)
Production Engineering, 42-63.

Bana e Costa, C. A. (1993). “Três convicções fundamentais na prática do apoio à decisão”. Pesquisa
Operacional, 13, 1-12.

Bana e Costa, C. A.; De Corte, J.M. & Vansnick, J.C. (2005). On the mathematical foundations of
macbeth. In: (edited by Greco, J. F. & Ehrgott, S. M.), Multicriteria Decision Analysis: state of
the art survey, 409-442, Springer Verlag, Boston, Dordrecht, London..

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Bana e Costa, C. A.; Ensslin, L.; Corrêa, É. C. & Vansnick, J. C. (1999). “Decision support systems in
action: integrated application in a multicriteria decision aid process”. European Journal of
Operational Research, 113, 315-335.

Barzilai, J. (2001). “On the Foundations of Measurement”. Proceedings of the 2001 IEEE
International Conference on Systems, Man and Cybernetics, Tucson, 7-10 October 2001.

Eden, C. & Ackermann, F. (1992). “The analysis of cause maps”. Journal of Management Studies, 29,
309-324.

Ensslin, L.; Dutra, A. & Ensslin, S. R. (2000). “MCDA: a constructivist approach to the management
of human resources at a governmental agency”. International Transactions in Operational
Research, 7, 79-100.

Ensslin, L.; Montibeller, G. N. & Noronha, S. M. (2001). “Apoio à Decisão: Metodologias para
Estruturação de Problemas e Avaliação Multicritério de Alternativas”. Insular, Florianópolis,
140-143.

Goodwin, P. & Wright, G. (1998). “Decision Analysis for Management Judgment”. John Wiley &
Sons, Chichester, 1-14.

Keeney, R. L. (1992). “Value-focused thinking: a path to creative decisionmaking”. Harvard

University Press, London, 3-154.

Landry, M. (1995). “A note on the concept of problem”. Organization Studies, 16, 315-343.

Roy, B. (1994). “On operational research and decision aid”. European Journal of Operational
Research, 73, 23-26.

Roy, B. (1996). “Multicriteria Methodology for Decision Aiding”. Kluwer Academic Publishers,
Dordrecht, 1-17.

Roy, B. (2005). “Paradigms and Challenges, Multiple Criteria Decision Analysis – State of the Art
Survey”. In: (edited by Greco, J. F. & Ehrgott, S. M.), Multicriteria Decision Analysis: state of
the art survey, 03-24, Springer Verlag, Boston, Dordrecht, London.

Roy, B. & Bouyssou, D. (1993). “Decision-aid: an elementary introduction with emphasis on multiple
criteria”. Information Science and Technology, 2, 109-123.

Roy, B. & Vanderpooten, D. (1996). “The european school of MCDA: emergence, basic features and
current works”. Journal of Multi-Criteria Decision Analysis, 5, 22-38.

Skinner, W. (1986). “The productivity paradox”. Management Review, 75, 41-45.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand