Int. J. Project Organisation and Management, Vol. 2, No.

4, 2010 361

A case study for project and portfolio management

information system selection: a group AHP-scoring
model approach

Vassilis C. Gerogiannis* and Panos Fitsilis

Department of Project Management,
Technological Research Centre of Thessaly,
Technological Education Institute of Larissa,
41110, Larissa, Greece
E-mail: gerogian@teilar.gr
E-mail: fitsilis@teilar.gr
*Corresponding author

Dimitra Voulgaridou
Department of Mechanical Engineering,
Sector of Industrial Management and Operational Research,
National Technical University of Athens,
Iroon Polytechniou 9, 15780 Athens, Greece
E-mail: dvoulg@central.ntua.gr

Konstantinos A. Kirytopoulos
Financial and Management Engineering Department,
University of the Aegean,
Kountourioti 41, 82100 Chios, Greece
E-mail: kkir@central.ntua.gr

Evi Sachini
Strategic Planning and Development Department,
Hellenic National Documentation Centre,
National Hellenic Research Foundation,
Vas. Constantinou 48, 11635 Athens, Greece
E-mail: esachin@ekt.gr

Abstract: Features variety, variations among organisations’ needs and the

plethora of powerful project and portfolio management (PPM) information
systems in the market, make the selection process among various PPM
information systems a complicated process. The large number of the evaluation
criteria involved during the decision aggravates the problem. This paper
addresses the limitation of the number of criteria in such complex decision
problems. It develops a hybrid model that exploits the benefits of the group
analytic hierarchy process (GAHP) along with the simplicity of the scoring
model for justifying the final selection. The proposed approach permits
consideration of much more criteria than the typical AHP approach can

Copyright © 2010 Inderscience Enterprises Ltd.

362 V.C. Gerogiannis et al.

reasonably handle and takes into account experts’ opinions for the comparison
of the PPM information systems as well as users’ opinions on specific
preferences/ needs. The proposed approach is applied on a real case in a public
Greek organisation.

Keywords: project and portfolio management information systems;

multi-criteria decision-making; MCDM; group decision-making; analytic
hierarchy process; AHP; hybrid model.

Reference to this paper should be made as follows: Gerogiannis, V.C.,

Fitsilis, P., Voulgaridou, D., Kirytopoulos, K.A. and Sachini, E. (2010) ‘A case
study for project and portfolio management information system selection:
a group AHP-scoring model approach’, Int. J. Project Organisation and
Management, Vol. 2, No. 4, pp.361–381.

Biographical notes: Vassilis C. Gerogiannis holds a Diploma and a PhD in

Software Engineering from the University of Patras, Greece. At present, he is a
Fulltime Assistant Professor at the Project Management Department in the
Technological Educational Institute (TEI) of Larissa, Greece. He is also a
Lecturer at the Greek Open University and a Module Leader in a Post Graduate
Program of Studies in Management Science, organised by TEI of Larissa in
cooperation with the Staffordshire University, UK. He has been participating as
a Technical Consultant and Project Manager in EU funded R&D projects. Up
to now, he has published more than 40 papers in international journals and
conference proceedings. His research interests include project management,
software project management and software engineering.

Panos Fitsilis is a Fulltime Professor at the Department of Project Management

of the Technological Education Institute of Larissa, Greece. He holds a
Diploma and a PhD from the Department of Computer Engineering and
Informatics from the University of Patras, Greece. He has extensive experience
in managing IT projects for large organisations of the private and public sector.
His research interests include project management methodologies, project
software management, software development methodologies and business
information systems.

Dimitra Voulgaridou holds an Electrical and Computer Engineer Diploma and

a PhD on Decision Support and Supply Chain Management. She works as a
Research and Teaching Associate at National Technical University of Athens
and at University of the Aegean. Her main research interests cover decision
support, supply chain management, project management and ERP systems.

Konstantinos A. Kirytopoulos holds a Mechanical Engineering Diploma

(Bachelor plus MSc equivalent) and PhD on Project Risk Management. He is
specialising in industrial management and is currently an Assistant Professor at
the University of the Aegean in Greece. His main research activity and
publications are in the fields of risk management, project management and
decision-making. More information about his academic activity may be reached
at www.kirytopoulos.eu.

Evi Sachini has been the Head of the Strategic Planning and Development
Department of the National Documentation Centre of Greece since 1997. She
holds a PhD in Chemical Engineering from the National Technical University
of Athens. She has been the manager of the project ‘National Information
System on Research and Technology’ aiming to develop open access
infrastructures addressed to the academic and research communities in Greece.
 A case study for project and portfolio management information system 363

1 Introduction

The exploitation of an appropriate project and portfolio management (PPM) information

system provides a lot of advantages for an organisation that undertakes projects and/or
portfolios of projects to implement organisational changes, business process redesign and
develop new products and/or services. A recent survey (Raymond and Bergeron, 2008)
demonstrates that project managers are starting to realise the benefits obtained from PPM
information systems, in terms of achieving increased productivity, effectiveness and
efficiency of their managerial tasks. Other empirical studies (Liberatore and
Pollack-Johnson, 2003) report that larger and more complex projects are pushing project
managers to use PPM software more. Contemporary PPM information systems have
embarked from single-user/single-project management systems to distributed, multi-user,
multi-functional software packages which offer integrated project management solutions,
not limited to scope management, budget and planning control. Market analyses in the
area (Wang, 2007; Light and Stang, 2008) present that modern PPM information systems
are trying to offer support for all knowledge areas described in the ‘Project Management
Body of Knowledge’ of the Project Management Institute (PMI, 2008) through covering
an expansive view of the ‘integration management’ knowledge area that includes
investment analysis, alignment of strategic programmes and support for business case
creation. Commercial PPM information systems provide support for a range of an
organisation’s activities by offering a number of features such as time, resource and cost
management, workgroup capabilities, reporting features, risk and contract management
support. More interested readers, not familiar with the technical characteristics of
commercial PPM systems, are referred to Ahlemann (2007), where detailed information
is given for 34 commercial PPM systems. In that report, each PPM system is presented
based on approximately 70 functional and 30 non-functional features.
This variety of features along with the variation among each organisation’s needs and
the plethora of PPM information systems in the market make the selection process of a
PPM system very difficult and complicated. The problem is often approached by ad hoc
procedures based only on personal preferences of the decision-maker or marketing
information/hype of PPM information systems and this may lead to a final selection that
does not precisely reflect the actual organisation’s needs or, even worse, to an unsuitable
PPM system. Empirical research shows that project managers often experience
limitations when using commercial project management software and may find it
inadequate for managing complex projects (White and Fortune, 2002). Therefore, a
well-established and systematic technique, from the multi-criteria decision-making
(MCDM) domain, can be useful to support a PPM system selection process. Although,
there is no a generic methodology that can be adopted for selecting a software package of
any type, literature reviews on evaluating software products suggest that users and
decision-makers can receive a lot of support, if they decide to adopt an MCDM method
(Jadhav and Sonar, 2009).
In particular, the findings of review studies (Vaidya and Kumar, 2006; Jadhav and
Sonar, 2009) present that the analytic hierarchy process (AHP) has been widely and
successfully used in evaluating several types of software packages (e.g., MRP/ERP
systems, simulation software, CAD/CASE systems, KM systems, etc.). The AHP method
364 V.C. Gerogiannis et al.

was introduced by Saaty (1980) and its primary objective is to classify a number of
alternatives (e.g., candidate software packages) by considering a given set of qualitative
and/or quantitative criteria, according to pairwise comparisons/judgements provided by
the decision-makers. AHP application results in a hierarchical levelling of the selection
criteria, where the upper hierarchy level is the goal of the decision process, the next level
defines the selection criteria, which can be further subdivided into subcriteria at lower
hierarchy levels and finally, the bottom level presents the alternative decisions to be
evaluated.
Although, AHP has been extensively used for the evaluation of various software
products and systems, little work has been done in the field of evaluating PPM
information systems. This can be attributed to the fact that despite the many advantages
of the AHP, scholars seem to agree that its main limitation is the large number of
pairwise comparisons needed, which result in significant time and effort requirements.
The time needed increases geometrically with the increase of the criteria involved,
making AHP practical prohibitive for complicate decision environments, such as the
selection of a PPM system.
In addition, PPM information systems selection, as every complicated problem, is
recommended to be held by groups rather than individuals (Tindale, 2003). This is
mainly because groups can represent a larger and more diverse set of perspectives and
constituencies, thus being more ‘fair’ (Tyler and Smith, 1998). Moreover, the idea that
‘two heads are better than one’ is widespread and typically accurate, based on the
empirical record (Bonner, 2004).
The research question for this paper is how can a decision-making process overcome
the limitation of a large number of criteria in a complex decision problem, such as the
selection of a PPM information system. As a response, in this paper a practical and easy
to use hybrid model for the evaluation of a set of PPM systems is developed, to help an
organisation to select a PPM system that reflects the organisation’s requirements. The
proposed approach exploits the benefits of MCDM and, in particular, encompasses all the
advantages of the AHP method and group-based AHP decision techniques, while it
manages to maintain the simplicity required by integrating the scoring model for the
evaluation of the criteria. Finally, the involvement of both users and experts in the
decision process achieves to exploit the expertise/interests of each one of them and to
strengthen the results. In a nutshell, the contribution of the proposed approach focuses on
a twofold innovation:

• it uses a hybrid model that permits consideration of much more criteria than the
typical AHP approach can reasonably handle

• it takes into account experts’ opinion for the comparison of the PPM information
systems as well as users’ opinion on specific preferences/needs.

The rest of this paper is organised as follows. Section 2 presents the relevant literature
background for AHP and group-based decision techniques. Software package selection
problems and existing PPM evaluation frameworks are also critically discussed. Section 3
provides a detailed description of the proposed hybrid model. Section 4 describes the
model application in a real case study, while conclusions and possible extensions of the
research work are addressed in Section 5.
 A case study for project and portfolio management information system 365

2 Literature review

2.1 MCDM and AHP applications in software package selection problems

An MCDM method (like AHP) overcomes some limitations of a conventional weighting
scoring method (WSM). On one hand, in a WSM, criteria weights and rating scales are
assigned arbitrary and real numbers are produced as final results. So, final comparative
values are often realised to represent the true difference among the decision alternatives,
rather than their relative ranking. Furthermore, assigning representative criteria weights
can be a very difficult task, when the number of selection criteria becomes large (Kontio,
1996).
On the other hand, among the main advantages of applying the AHP method one
could consider that (Jadhav and Sonar, 2009):
1 it is capable to provide a hierarchical decomposition of a decision problem that helps
in better understanding of the overall decision-making process
2 it handles both quantitative and qualitative criteria
3 it is based on relative, pairwise comparisons of all decision elements; instead of
arbitrarily defining a percentage score and a weight for each decision element, AHP
allows the decision-maker to focus on the comparison of two criteria/alternatives, at
a time, thus, it decreases the possibility of defining ratings based only on personal
perceptions of the evaluators or other external influences
4 AHP is applicable to both individual and group-based decision-making (this is often
achieved by considering the geometric mean of all comparison values)
5 it enables consistency checks upon pairwise decision-comparison judgements
6 it supports sensitivity analysis to examine the effects of changing values of criteria
weights on the final ranking of the decision alternatives
7 the method applicability and its extended features (e.g., sensitivity analysis) are
supported by software systems – e.g., expert choice (www.expertchoice.com), super
decisions (www.superdecisions.com) – although, pairwise comparisons and
calculations/consistency checks can be easily implemented in a spreadsheet
application software.
All the above advantages have influenced the wide application of AHP to MCDM
problems, in many different sectors, including project management and software
engineering project management. As far as project management is concerned, Al Harbi
(2001) presented a field survey of applications of AHP, focusing, in particular, on the
selection problem of the best contractor to undertake a construction project. The case
study in this research uses a group decision-making approach that involves characteristics
of candidate contractors as well as project priorities defined by the project owner.
Prioritising software requirements and selecting component off the self-systems (COTS)
are two representative examples of software engineering project management problems
which have gained attention to be supported by AHP. In these problems, AHP has been
used to compare software requirements (Karlsson and Ryan, 1997) or COTS products
(Kontio, 1996; Lozano-Tello and Gomez Perez, 2002) by taking into account the relative
importance between value and cost of each requirement/COTS product.
366 V.C. Gerogiannis et al.

The application of AHP has been reported in many other case studies related to
software package selection problems. The interested reader is referred, e.g., to (Ahmad
and Laplante, 2007) where the authors propose the use of AHP to evaluate three imaging
software packages by combining seven types of subjective data (software complexity
metrics) and experts’ opinions. Their approach helps in deciding an appropriate imaging
software when there are contradictory complexity metrics (quantitative data) for each
candidate software package, by incorporating experts’ judgements and relying on their
experience within the subject. Another representative example is the application of AHP
in the selection of a suitable knowledge management (KM) tool (Ngai and Chan, 2005).
The authors list various candidate decision criteria and subcriteria for justifying the
selection process but, in their case study, they actually show the use of only three major
criteria (cost, functionality and vendor) to evaluate three alternative KM systems.
Considering additional criteria or subcriteria, often leads to a combinatorial explosion of
the number of individual pairwise comparisons, thus, making AHP application very time
and cost ineffective (Maiden and Ncube, 1998). A similar problem appears in another
approach that follows AHP for the selection of an enterprise resource planning (ERP)
information system (Karaarslan and Gundogar, 2008). The authors suggest eight selection
criteria (seven of them correspond to different modules of an ERP system and the ninth
criterion covers the general system functionality) and a very detailed list of subcriteria for
the evaluation of two ERP systems. The evaluators in the presented case study were
asked to rate explicitly (with an integer value from zero to five) each subcriterion, an
approach that improved the practical application of the decision-making method.
Despite its popularity, there are two main difficulties related with the practical
application of AHP. First, when determining ‘crisp’ comparative values, any
uncertainties on judgements of decision-makers cannot be easily handled (Saaty, 1980).
A proposed solution for this limitation is to use a combination of AHP with fuzzy logic
(Chang, 1996). For example, in Cebeci (2009), a combined fuzzy logic – AHP based
approach is proposed to select the most suitable ERP system for a textile manufacturing
company. However, the complexity of fuzzy logic approaches often raises difficulties for
their practical application; thus, the problem is practically handled with sensitivity
analysis. A second difficulty in AHP application appears when there are dependencies
among the selection criteria. In such a case, the analytic network process (ANP) should
be used (Kirytopoulos et al., 2008a). ANP is an AHP extension that handles both
intra- and inter-dependencies among clusters of selection criteria (Saaty, 1996). The
application of the ANP method can be, however, very time and cost intensive, since it
often requires a great number of pairwise comparisons (Voulgaridou et al., 2009). For
example, a case study of ANP use in selecting among three ERP systems, with respect to
12 selection criteria organised into system and vendor related clusters, concludes that
considering all possible interactions among criteria requires much more additional time
and effort (Percin, 2008) than the typical AHP approach.
Although, AHP has been extensively used for the evaluation of various software
products, surprisingly little work has been done in the field of evaluating PPM
information systems with AHP or even by considering its extensions, such as ANP. Even
those who tried to use the method for project management systems selection (Ahmad and
Laplante, 2006) admitted that their work is rather indicative with main objective to
expose a representative case for illustrating the selection process and not to create a
definitive set of criteria/features that should be taken into account in practice. The
reasoning behind this shortcome in bibliography should be probably assigned to the
 A case study for project and portfolio management information system 367

inability of the AHP and ANP methods to handle problems with many criteria. Selecting
a sophisticated PPM information system may entail more than 100 criteria/functional
characteristics (Ahlemann, 2007; NASA PMTWG, 2004). Thus, the use of a hybrid
model such as the one proposed in this paper, could be efficient for practically using AHP
in a real practical context.
On the other hand, the selection of factors involved in decision-making for selecting a
PPM information system is crucial for an effective outcome. Sophisticated PPM
information systems no longer emphasise only on scheduling and resource management
issues. They are software tools that can support the life-cycle management of a project or
a portfolio of projects. Therefore, the selection of an appropriate PPM system is often a
complex process that requires systematic support. There are research papers in the
relevant literature which aim to compare PPM information systems through the
evaluation of quality features [such as user friendliness and documentation; De Wit and
Herroelen (1990)] or conventional technical characteristics, such as scheduling and
resource management capabilities (Rushinek and Rushinek, 1991; Maroto and Tormos,
1994). However, it seems to be a lack of comparative approaches that consider both
experts’ judgements and PPM system users’ requirements, in an actual organisational
context. In a real world context, it should be stressed that the users should decide on what
is important or not for their work and experts decide on which PPM information system
is better for each criterion addressed.
The proposed approach in this paper follows a practical, hybrid, group-based method
that was applied in an actual context of an organisation that required support for selecting
an appropriate PPM system to fulfil specific project/portfolio management requirements.
The selection criteria were specified according to a comprehensive, functional oriented
evaluation framework (NASA PMTWG, 2004). The result was a detailed set of
criteria/functional features grouped into clusters.

2.2 Group analytic hierarchy process (GAHP)

Group decision-making is considered to be a process for deriving a single group
preference from a number of individual preferences regarding a finite set of criteria and
alternatives. The group decision-making process in the AHP involves the construction of
pairwise comparison matrices at each level of network either by consensus voting or by
aggregating the individual preferences (Saaty, 1989). Forman and Peniwati (1998)
showed that the group prioritisation methods in the AHP method apply two basic
techniques for aggregating the individual preferences into a group preference, depending
on whether the group wants to act together as a unit or as separate individuals, and
specify two aggregation approaches:
1 aggregating individual judgements (AIJ)
2 aggregating individual priorities (AIP).
AIJ is a synergistic aggregation approach, where the judgements of the decision-makers
for each set of pairwise comparisons are aggregated into a new set of aggregated group
judgements at each level. The aggregated group judgements are considered as judgements
of a ‘new individual’ and the priorities of this individual are derived as a group solution.
In the AIP approach, the group is considered as a collection of individuals, with different
value systems. This approach requires priorities to be obtained from each individual, and
368 V.C. Gerogiannis et al.

then aggregated into final group priorities. Both aggregation approaches in the AHP
involve two independent stages, performed in different successions: group synthesis and
prioritisation (Bryson and Joseph, 1999). Group synthesis is based on an additional
aggregation procedure that combines the individual judgements (AIJ) or priorities (AIP)
into group preferences, while the prioritisation stage requires an appropriate prioritisation
method for deriving a single group priority (AIJ) or a number of individual priorities
(AIP).
The weighted arithmetic mean (WAM) and the geometric mean (GM) mathematical
procedures are commonly used to determine group preferences for both the AIJ and AIP
aggregation approaches. Aczel and Saaty (1983) claim that only the GM is an appropriate
procedure as it preserves the reciprocal properties of the aggregated pairwise comparison
matrices. Forman and Peniwati (1998) also state that the GM should be used for AIJ, as
for AIP either the WAM or GM are meaningful procedures, satisfying the Pareto
optimality principle. Lately, a third approach, the weighted geometric mean (WGM) is
considered to be the optimal procedure (Levy and Taji, 2007). To illustrate the group
prioritisation process in the AHP, let aij,z represent the comparison of elements i and j,
with respect to the element z, in a pairwise comparison matrix A for n decision-makers.
The group judgement by taking the weighted geometric mean of all comparisons is equal
to:
1
n p
∑ aij ,iz ⎛ n ⎞
⎛ n ⎞
i=1 ⎜
⎜
∑ p ln a i ij , z ⎟
⎟
wij , z = ⎜
⎜ ∏a pi
ij , z ⎟
⎟
= exp ⎜ i =1
n ⎟
⎝ i =1 ⎠ ⎜
⎜
⎝
∑p
i =1
i
⎟
⎟
⎠
where aij,z is each person’s paired comparison, n is the number of decision-makers, and pi
(0, 0 ≤ pi ≤ 1, 0) is the importance priority for the person’s paired comparison.

2.3 Evaluation frameworks for PPM information systems

Raymond and Bergeron (2008) reported that a number of project managers indicate
strong impact of the PPM system upon the successful completion of their projects, while
others do not. Their research indicates that ‘unsatisfied’ project managers are depended
upon a PPM system that produces lower quality information; hence they use the system
less and they are less supported in their project management tasks. From this research, we
can conclude that it is important for an organisation to select a proper PPM system that
covers technical, managerial and organisational needs.
The PPM system selection process can be supported by referencing to
published survey results. In the past, e.g., PMI had published an extensive survey (PMI,
1999) that compared more than 200 different PPM information systems in dimensions
like scheduling, cost management, risk management, resource management and
communication management. However, such detailed comparisons focus rather on
technical factors which represent vendors’ perspectives and they should be always
utilised with care by considering specific project management needs within the context of
individual organisations. Support in setting up a PPM system can be also gained by
considering the users’ perceptions from a PPM system usage. This often requires the
application of a technology acceptance model to provide some empirical evidence on the
 A case study for project and portfolio management information system 369

relationships that exist, e.g., between a system’s usefulness and its ease of use (Davis,
1989). A representative example of this approach is the one presented in (Ali and Money,
2005). This work surveyed 497 PPM system users and the final result was a general index
for measuring the effectiveness of PPM information systems according to four,
user-oriented, dimensions (i.e., information quality, system functionality, ease of use,
performance impact). However, the theoretical background of such an index lacks the
mathematical formality of an MCDM approach. Formal support can be proven useful in
cases that selection criteria contradict each another and respective PPM system users are
actually expressing their preferences and not their actual knowledge on potential benefits
derived from a PPM system.
To reach a more detailed list of selection factors, evaluators-decision-makers may
utilise criteria offered by a conceptual reference information model for PPM information
systems, like e.g., the RefModePM model (Ahlemann, 2009). A conceptual reference
model provides a holistic approach for a PPM system selection process, since it handles
the selection problem from a business process reengineering viewpoint. A reference
model can be followed, first of all, to categorise all respective PPM system users (i.e.,
members of an organisation personnel), then to identify the organisation’s project
management processes and, finally, to provide descriptions for the organisation’s project
data. RefModePM is mainly a process oriented framework that classifies project
management processes within a project-based organisation into the following general
areas, which, in a broad sense, follow the phases of a project life-cycle:
1 project idea generation
2 project idea evaluation
3 portfolio planning
4 project preparation
5 detailed planning
6 project execution
7 project controlling
8 portfolio controlling
9 internal project termination
10 external project termination.
Each of these areas is further subdivided into specific process steps which finally, form
the selection criteria for a candidate PPM system.
Apart from RefModePM, to decide upon PPM information systems selection criteria,
we have considered another evaluation framework, proposed by the NASA PPM
information systems working group (NASA PMTWG, 2004). This framework
(NASAPM) is more oriented towards functional features of a PPM system. It lists
explicitly a number of technical/performance/reporting requirements to be used as
selection criteria, where each requirement is further analysed into a set of functional
features. The following nine evaluation clusters of criteria, based on the NASAPM
evaluation framework, have been used in our selection process:
370 V.C. Gerogiannis et al.

C1 open database connectivity and architecture – workgroup and networking capabilities

C2 ease of use
C3 project scheduling support – definition of project task/field features

C4 baselining – tracking project progress and calendar features

C5 resource and cost management features
C6 risk management features
C7 reporting features
C8 support for supplier and contract management
C9 portfolio management features.

Each cluster includes a set of specific criteria/functional features. For instance, cluster C3
(project scheduling support – definition of project task/field features) summarises 18
criteria/functional features, namely:
1 perform basic scheduling/PERT functionality
2 allow variable scaling for task duration
3 perform full critical path method (CPM) functionality, including capability of
showing multiple critical paths in output reports
4 allow users to designate logical relationships (i.e., start-to-start/SS,
start-to-finish/SF, finish-to-finish/FF, and finish-to-start/FS)
5 allow users to customise tables and views
6 allow users to create project templates
7 generate an organisational breakdown structure (OBS) and a work breakdown
structure (WBS) or allow user to impose a WBS
8 allow users to assign positive or negative lag/lead times on logical relationships
9 perform resource levelling and smoothing
10 have the capability of ‘de-linking’ percent complete from remaining duration
11 allow users to define and assign constraints to tasks-milestones
12 allow users to specify tasks or milestones to be rolled-up
13 allow users to define fields for each project/task/resource
14 roll-up multiple projects into a master schedule
15 define a task with the duration being automatically calculated based upon its
dependency with another task (hammock task)
 A case study for project and portfolio management information system 371

16 incorporate a large comment/notes field for entry of soft information

17 define task start/end dates as fixed, resource-driven or effort-driven

18 allow users to create a read-only version of project (fields, tables, resources,

calendars).
In total, for the nine listed above, evaluation clusters, a total number of 105 criteria have
been identified to be evaluated.
This vast number of criteria prevents decision-makers from using the typical AHP
approach. It should be mentioned here that the use of ANP is not considered at all, first
because there are no dependencies among criteria of different clusters (the problem can
be indeed modelled as a hierarchical problem) and second because its use would be
even more complicated than the AHP for this number of criteria (Kirytopoulos et al.,
2008b).

3 The proposed hybrid model

The proposed framework consists of a ‘four-steps’ hybrid model, as shown in Figure 1.

The first step employs the GAHP method in order that the users classify the clusters of
the criteria based on their relative importance. The second step is performed by the
experts who, based on the scoring model, assign weights to the specific criteria/functional
features of the PPM information systems. Then, the overall importance of each
alternative is calculated by aggregating the results of the GAHP and the scoring model.
Finally, sensitivity analysis is performed in order to evaluate the stability of the final
decision. A detailed description of each step follows hereafter:
Step 1 The users evaluate the clusters of the criteria by implementing the AHP
according to the needs of their organisation. The users are divided into two
groups. Team I users’ group consists of project managers involved in planning,
executing and monitoring of single projects and Team II users’ group consists of
project officers, responsible for strategic management, contract management,
multi-project coordination and planning the organisation’s project portfolios.
Each team assigns weights to the clusters according to its value system and the
weighted geometric mean is utilised in order to calculate final weights of this
step. As far as it concerns the clusters, they are adopted from the NASAPM
evaluation framework:
C1 open database connectivity and architecture – workgroup and networking
capabilities
C2 ease of use
C3 project scheduling support – definition of project task/field features
C4 baselining – tracking project progress and calendar features
C5 resource and cost management features
C6 risk management features
C7 reporting features
372 V.C. Gerogiannis et al.

C8 support for supplier and contract management

C9 portfolio management measures.
The aggregation of the two teams’ judgements is performed using the GAHP in
order to obtain the final weights of the clusters according to the users.
Step 2 The experts implement the scoring model in order to evaluate the alternatives
(PPM information systems) with respect to each of the criteria. The criteria
selected correspond to the functional features, as proposed by the NASAPM
evaluation framework.
Step 3 The weights of the decision-makers (both users and experts) are aggregated in
order to provide the final result. The following formula describes the
mathematical expression of that aggregation:

WN = wc1 × N c1 + wc 2 × N c 2 + ... + wc9 × N c 9 (1)

where
WN the final weighted score of the alternative N
Wci the weight of cluster i (i = 1, 2, ..., 9) according to users’ judgements, and
Nci the score of the alternative N for the criteria included in cluster i (i = 1, 2, …, 9)
according to the experts.
Step 4 This final step performs sensitivity analysis to the final weights obtained during
the previous step in order to determine if the final ranking of the alternatives is
stable to changes in the inputs (judgements or weights).
The following section applies the proposed hybrid model to a real case of selecting a
PPM system for satisfying the needs of a public organisation in Greece that undertakes
projects on science research and technology issues.

Figure 1 The proposed hybrid model (see online version for colours)
 A case study for project and portfolio management information system 373

4 Case study

4.1 General description

This section presents the application of the proposed hybrid GAHP-scoring model within
the context of a project that took place in 2008 with the overall goal of selecting the most
appropriate PPM system for the Hellenic National Documentation Centre (NDC)
(www.ekt.gr). NDC is the public organisation in Greece responsible for providing
documentation and information support on science research and technology issues. Since
its establishment, in 1980, NDC has been involved in a number of projects, funded by
national and EU operational programmes. The organisation operates the National Science
and Technology Digital Library, develops digital content (such as the National Archive
of PhD theses), supports the networking of the Hellenic University Digital Libraries and
operates as national contact point for some EU funded R&D programmes like ‘ideas’,
‘cooperation’ and ‘capacities’.
NDC does not maintain an integrated project/portfolio management infrastructure. A
team of 16 employees (users) with project management responsibilities undertakes and
manages the organisation’s projects. Each of these employees utilises a stand-alone PPM
system according to his/her own personal preferences/experience. In order to increase
project management effectiveness and productivity, NDC has decided to adopt a
collaborative PPM information system. NDC appointed the Technological Research
Center of Thessaly in Greece to act as an expert and aid the decision-making process.
Three experts (evaluators) from the Technological Research Center, with an average of
five years teaching/professional experience in PPM systems, were involved in this
process, aiming to identify NDC requirements from a PPM system and to select an
appropriate system that will cover these requirements.

4.2 Implementation of the proposed hybrid model

Model construction
Three joint meetings between the users and the experts group were conducted to agree
upon selection criteria and candidate PPM information systems. In the first meeting, the
experts group from the Technological Research Center of Thessaly presented to
the users group from the NDC the details of the NASAPM systems evaluation
framework. Then the users and the experts agreed to base the selection process on
technical/performance/reporting requirements listed in the NASAPM framework. In the
second meeting, the users were asked to provide answers to a detailed questionnaire with
the aim to identify the specific project management responsibilities of each user. Through
analysing their answers, two teams of users were identified: Team I users group that
consisted of 11 project managers involved in planning, executing and monitoring of
single projects and Team II users group that included five project officers, responsible for
strategic management, contract management, multi-project coordination and planning the
organisation project portfolios. In the third meeting, a discussion took place upon
possible candidate PPM information systems to be evaluated. Though there is a large
number of an available PPM information system, both experts and users were queried to
express their general opinion on ten PPM information systems which in market survey
results (Light and Stang, 2008) are characterised as leaders and challengers in this
374 V.C. Gerogiannis et al.

segment of enterprise software market. Five from these PPM information systems were
excluded for two reasons. First, since they do not have presence in the national market
and second, because experts were persuaded that their usage was inappropriate for the
specific case, mainly due to lack of technical support and the non-availability of training
services. This first-level screening resulted in a list of five powerful, widespread PPM
information systems with portfolio management and workgroup capabilities and strong
presence (i.e., technical/training support) in the national market. For project
confidentiality reasons and aiming at the non-commercial promotion of any software
package, we will refer to these PPM information systems as A, B, C, D and E. Based on
the above, the decision model was constructed, as shown in Figure 2.

Figure 2 Hierarchical structure of the PPM information systems selection problem

Step 1 Users’ evaluation on clusters

After the construction of the model, the first step of the proposed hybrid model,
concerning the evaluation of the clusters, was implemented. To analyse the users’
requirements from a PPM system, the authors designed and disseminated to the two
teams of the NDC personnel an AHP-structured questionnaire, asking from each one to
comparatively evaluate the nine clusters and express their perception on the relative
importance of each cluster with respect to the overall performance and the benefits
provided from a candidate PPM information system. Each of the 16 users was requested
to perform 36 pairwise comparisons. The judgements of the 11 members of Team I
(project managers) were aggregated using the geometric mean in order to obtain the final
group decision result. Similarly, the geometric mean was implemented in order to obtain
the group weight of Team II. The final clusters’ weights were calculated using the
weighted geometric mean (WGM) as shown in formula 2:
 A case study for project and portfolio management information system 375

⎛p TeamI
ln wCi TeamII
+ pTeamII ln wCi ⎞
wCi = exp ⎜ TeamI ⎟⎟ (2)
⎜ pTeamI + pTeamII
⎝ ⎠
where, Wci, the final weight of the cluster i (i = 1, 2, ..., 9), WciTeamI and WciTeamII, the
weights of the cluster i according to Teams I and II, respectively and pTeamI and pTeamII, the
importance priorities of Team I and Team II, respectively. For the present case the
importance priorities were assigned as pTeamI = 0.6 and pTeamII = 0.4. These priorities
comprise a strategic decision by the senior management that reflect its perception about
the importance of each group (team) towards the final decision-selection. The final
weights, as obtained using the WGM, are presented in Figure 3.

Figure 3 Final weights of clusters (see online version for colours)

Step 2 Experts’ assessment on PPM information systems

The final model, as described above, resulted in the identification of five candidate PPM
information systems and nine clusters which include 105 criteria (functional features of
the NASATM evaluation framework). In order to evaluate the candidate systems in a
manageable and reliable way, the experts scored the performance of the PPM information
systems with respect to the criteria previously identified. In particular, they utilised an
ordinal scoring scale, similar to the one proposed by RefModePM (Ahlemann, 2007).
The three experts collectively evaluated each PPM system by providing a general
accepted score value on each one of the 105 functional features and they utilised a
‘two-stars’ score (zero-stars: no functionality, one-star: adequate functionality, two-stars:
advanced functionality). For example, considering the feature ‘perform resource levelling
and smoothing’ of cluster C3 (‘project scheduling support – definition of project
task/field features’), all tools received by experts one star (adequate functionality), except
from tool B that scored two stars (advanced functionality). Experts in that case justified
their ranking as tool B was the only one that allowed the user to define an unlimited
number of priority rules for resource levelling. All experts provided a short written
justification for every score they gave. Any deviations/missing scores were resolved in a
joint meeting held by the three experts and a final consensus was reached on the
evaluation of each of the 105 functional features.
376 V.C. Gerogiannis et al.

The result of the experts’ assessment was an aggregate total score for each system, in
each one of the nine clusters. For example, according to cluster C3 (project scheduling
support – definition of project task/field features), an aggregate score of system A equal
to 21/36 means that the experts perceived that system A scores 21 stars in total of 36 stars
(C3 includes 18 features * 2 (maximum number of stars) = 36 stars).
The final scores, as they were given by the experts, for each system in each cluster are
shown in Table 1. Thus, Table 1 shows the degree to which each PPM system scores in
criteria of each cluster. For instance, for cluster C1 (open database connectivity and
architecture – workgroup and networking capabilities), system B ‘gathered’ 17 stars out
of 28.
The values of Table 1 were normalised (i.e., in [0...1] scale) by dividing each score by
its column total. The normalised matrix is presented in Table 2. This normalisation is
necessary in order for the final weighted score of each alternative to be attained. For
example, the values of column C1 in Table 1 are 0.75 (= 21/28); 0.61; 0.86; 0.68; 0.79
and its sum equals to 3.68. Thus, the normalised value for alternative A in Table 2 is
0.2039 (= 0.75/3.68). This table depicts the final normalised scores of all PPM
information systems according to experts’ judgements.
Table 1 Final scores given by the experts

PPM Clusters
information
systems C1 C2 C3 C4 C5 C6 C7 C8 C9
A =21/28 =5/20 =21/36 =13/24 =13/22 =9/14 =20/42 =3/12 =7/12
B =17/82 =13/20 =23/36 =14/24 =13/22 =2/14 =23/42 =4/12 =7/12
C =24/28 =12/20 =22/36 =8/24 =14/22 =10/14 =10/42 =6/12 =6/12
D =19/28 =10/20 =23/36 =13/24 =14/22 =10/14 =21/42 =4/12 =6/12
E =22/28 =10/20 =23/36 =12/24 =12/22 =10/14 =23/42 =7/12 =6/12

Table 2 Normalised scores of dPPM information systems

PPM Clusters
information
systems C1 C2 C3 C4 C5 C6 C7 C8 C9
A 0.2039 0.1000 0.1875 0.2167 0.1970 0.1915 0.2062 0.1250 0.2188
B 0.1650 0.2600 0.2054 0.2333 0.1970 0.1702 0.2371 0.1667 0.2188
C 0.2330 0.2400 0.1964 0.1333 0.2121 0.2128 0.1031 0.2500 0.1875
D 0.1845 0.2000 0.2054 0.2167 0.2121 0.2128 0.2165 0.1667 0.1875
E 0.2136 0.2000 0.2054 0.2000 0.1818 0.2128 0.2371 0.2917 0.1875

Step 3 Aggregation of users’ and experts’ judgements

In order to aggregate users’ and experts’ judgements into one final solution, the formula
(1) presented in Section 2 is utilised. For the alternative A the final weighted score is
calculated as follows:
WA = wc1 × Ac1 + wc 2 × Ac 2 + wc3 × Ac3 + wc 4 × Ac 4 + wc 5 × Ac5 + wc 6 × Ac 6 + wc 7 × Ac 7
(3)
+ wc8 × Ac8 + wc 9 × Ac 9
 A case study for project and portfolio management information system 377

where
WA, the final weighted score of the alternative A, wci, the weight of the cluster i
(i = 1, 2, ..., 9) according to users’ judgements (obtained from Figure 3), Aci, the score of
the alternative A for the criteria included in cluster i (i = 1, 2, …, 9) according to the
experts (obtained from Table 2). And by replacing values we get WA = 0,185.
The final weighted scores for all the alternatives are shown in Figure 4, from where it
can be deduced that alternative E is the most dominant PPM information system for the
present case study.

Figure 4 Final weighted scores of the alternatives (see online version for colours)

0.220
0.215
0.215
0.210
0.205 0.202
0.200 0.197 0.197
0.195
0.190
0.185
0.185
0.180
0.175
0.170
A B C D E

Figure 5 Final weights of the alternatives after sensitivity analysis (see online version for
colours)

Step 4 Sensitivity analysis

Sensitivity analysis is concerned with ‘what-if’ kind of questions to determine if the final
answer is stable to changes (experiments) in the inputs, either judgements or priorities. In
the present case, sensitivity analysis was performed by changing the priorities. Of special
378 V.C. Gerogiannis et al.

interest is to see if these changes alter the order of the alternatives. In order to do so the
weights of all clusters have been changed by a factor of 30%. Figure 5 depicts the
ranking of the alternatives in a scenario where wc1 (the priority of the dominant cluster)
would be reduced by a factor of 30%. The graph shows that even during this extreme
scenario, information system E remains the dominant alternative. Performing similar
sensitivity tests for other clusters led to the conclusion that the outcome is very stable and
does not change the overall ranking despite that all the alternatives receive a very close
final weight.

5 Conclusions

This paper presented a hybrid model, which utilises both the GAHP and the scoring
method, as an answer to the basic research question provided in the abstract, namely
‘how can a decision-making process overcome the limitation of a large number of
decision criteria when selecting a PPM information system?’ The proposed hybrid model
is argued to be suitable for decision problems that comprise a significant number of
criteria (105 for the studied problem) and thus, cannot be handled with typical MCDA
methods as AHP because of the great number of pairwise comparisons needed. The
proposed method encompasses all the advantages of the GAHP, while overcoming its
major limitation concerning the time needed for the comparisons, as the scoring model
has been utilised for this purpose. The innovativeness of the proposed approach is
twofold. On one hand, it uses a hybrid model that permits consideration of much more
criteria than the typical AHP approach can reasonably handle and, on the other hand, it
takes into account both experts’ and user’s opinion for making the right decision. Experts
can utilise the scoring model to express their judgements on the selection
criteria/functional features of PPM systems, while users evaluate pairwise the selection
criteria with respect to what they perceive as more important for their project success.
Thus, the final selection is based on the aggregation of both experts’ and users’
judgements.
The method presented here, through discussing a real case study, exposes how a
group-based, multi-criteria decision analysis approach can be applied to facilitate the
selection problem of an appropriate PPM system. All problem stakeholders (i.e., both
PPM system experts and PPM system users) have been actively involved in the selection
process, so that their expertise could be exploited at a maximum degree. Specifically, the
users provided the model with their corporate knowledge concerning the needs of their
organisation, while the experts fed the model with the knowledge concerning the
information systems functionality.
The results of the model (Figure 3) indicate that system E offers a powerful
environment with integrated support for both organisation resource planning and project
management services. System B encompasses a widespread project management system
that operates at an organisation-level platform, offering portfolio and multi-user
management capabilities. It appears that the least promising system is system A. System
A is a PPM information system specialised mainly in IT service portfolio and financial
management functionalities, with low technical support responsiveness in the national
market. As far as the evaluation of system C is concerned, its main advantageous
characteristic is the strong integration with database management applications (i.e., it
received, from the experts, the highest score in cluster C1 ‘open database connectivity
 A case study for project and portfolio management information system 379

and architecture – workgroup and networking capabilities’). Finally, one of the strongest
advantages of system D, which was weighted equal to C, is the support for quantitative
risk management techniques. Finally, the sensitivity analysis performed for the specific
case study implies that the output of the model is rather stable, since the ranking of the
alternatives is preserved.
This study raises several important issues that could spark further research. An
interesting idea could be the exploration of the sensitivity concerning the ‘star’ scoring
model. Treating with uncertainties (e.g., by applying a fuzzy AHP approach) would
strengthen the proposed approach in deriving more precise results. The authors also plan
to further validate the method’s implications and applicability in other case problems
addressing the selection of other types of software packages. Although not applicable in
the case study presented here, dependencies among criteria may be present in other cases,
thus such an issue could be handled by applying a more extended decision-making
technique (e.g., the ANP).

Acknowledgements

The authors would like to thank Mr. George Kakarontzas and Mr. Athanasios Tzikas for
being supportive in the application of the approach described in this paper. The authors
also acknowledge the enthusiastic involvement of the personnel of the Hellenic National
Documentation Centre in the empirical part of the research. Finally, the authors wish to
thank the editor and the anonymous reviewers for their constructive critique that helped
the improvement of this paper.

References
Aczel, J. and Saaty, T. (1983) ‘Procedures for synthesising ratio judgements’, Journal of
Mathematical Psychology, Vol. 27, No. 1, pp.93–102.
Ahlemann, F. (2007) Project Management Software Systems – Requirements, Selection Processes
and Products, A study by the European Business School, 5th ed., Würzburg, BARC.
Ahlemann, F. (2009) ‘Towards a conceptual reference model for project management information
systems’, International Journal of Project Management, Vol. 27, No. 1, pp.19–30.
Ahmad, N. and Laplante, P. (2006) ‘Software project management systems: making a practical
decision using AHP’, Proceedings of the 30th Annual IEEE/NASA Software Engineering
Workshop, pp.76–84.
Ahmad, N. and Laplante, P. (2007) ‘Reasoning about software using metrics and expert opinion’,
Innovations in Systems and Software Engineering, Vol. 3, No. 4, pp.229–235.
Al Harbi, K.M. (2001) ‘Application of AHP in project management’, International Journal of
Project Management, Vol. 19, No. 4, pp.19–27.
Ali, A.S.B. and Money, W.H. (2005) ‘A study of project management system acceptance’,
Proceedings of the 38th Hawaii International Conference on System Sciences, IEEE Press,
pp.234–244.
Bonner, B. (2004) ‘Expertise in group problem Solving: recognition, social combination and
performance’, Group Dynamics: Theory, Research, and Practice, Vol. 8, No. 4, pp.277–290.
Bryson, N. and Joseph, A. (1999) ‘Generating consensus priority point vectors: a logarithmic goal
programming approach’, Computers & Operations Research, Vol. 26, No. 6, pp.637–643.
380 V.C. Gerogiannis et al.

Cebeci, U. (2009) ‘Fuzzy AHP-based decision support system for selecting ERP systems in textile
industry by using balanced scorecard’, Expert Systems with Applications, Vol. 36, No. 5,
pp.8900–8909.
Chang, D.Y. (1996) ‘Application of extent analysis method on fuzzy AHP’, European Journal of
Operational Research, Vol. 95, No. 3, pp.649–655.
Davis, F.D. (1989) ‘Perceived usefulness, perceived ease of use and user acceptance of information
technology’, MIS Quarterly, Vol. 13, No. 3, pp.319–340.
De Wit, J. and Herroelen, W.S. (1990) ‘An evaluation of microcomputer-based software packages
for project management’, European Journal of Operational Research, Vol. 49, No. 1,
pp.102–139.
Forman, E. and Peniwati, K. (1998) ‘Aggregating individual judgements and priorities with the
AHP’, European Journal of Operational Research, Vol. 108, No. 1, pp.165–169.
Jadhav, A.S. and Sonar, R.M. (2009) ‘Evaluating and selecting software packages: a review’,
Information and Software Technology, Vol. 51, No. 3, pp.555–563.
Karaarslan, N. and Gundogar, E. (2008) ‘An application for modular capability-based ERP
software selection using AHP method’, The International Journal of Advanced Manufacturing
Technology, Vol. 42, Nos. 9–10, pp.1433–3015.
Karlsson, J. and Ryan, K. (1997) ‘A cost-value approach for prioritizing requirements’, IEEE
Software, Vol. 14, No. 5, pp.67–74.
Kirytopoulos, K., Leopoulos, V. and Voulgaridou, D. (2008a) ‘Supplier selection in pharmaceutical
industry: an analytic network process approach’, Benchmarking: An International Journal,
Vol. 15, No. 4, pp.494–516.
Kirytopoulos, K., Leopoulos, V. and Voulgaridou, D. (2008b) ‘Integrating multicriteria decision
making and multiobjective optimization methods for supplier selection’, Proceedings of the
15th International Working Seminar on Production Economics, Vol. 3, pp.211–222.
Kontio, J. (1996) ‘A case study in applying a systematic method for COTS selection’, Proceedings
of the 18th International Conference on Software Engineering, IEEE Press, pp.201–209.
Levy, J. and Taji, K. (2007) ‘Group decision support for hazards planning and emergency
management: a group analytic network process (GANP) approach’, Mathematical and
Computer Modelling, Vol. 46, Nos. 7–8, pp.906–917.
Liberatore, M.J. and Pollack-Johnson, B. (2003) ‘Factors influencing the usage and selection of
project management software’, IEEE Transactions on Engineering Management, Vol. 50,
No. 2, pp.164–174.
Light, M. and Stang, D.B. (2008) ‘Magic quadrant for IT project and portfolio management’,
Gartner RAS Core Research Note, G00157924, Gartner Research.
Lozano-Tello, A. and Gomez Perez, A. (2002) ‘BAREMO: how to choose the appropriate software
component using the analytic hierarchy process’, Proceedings of the International Conference
on Software Engineering and Knowledge Engineering (SEKE), pp.781–788.
Maiden, N.A. and Ncube, V. (1998), ‘Acquiring COTS software selection requirements’, IEEE
Software, Vol. 15, No. 2, pp.45–56.
Maroto, C. and Tormos, P. (1994) ‘Project management: an evaluation of software quality’,
International Transactions in Operational Research, Vol. 1, No. 2, pp.209–221.
NASA PMTWG (2004) NASA Project Management Tool Analysis and Recommendations White
Paper, Project Management Tool Working Group, Principal Center for Workgroup HW and
SW, NASA Glenn Research Center, available at km.nasa.gov/pdf/54927main_pm-tool-
paper.pdf.
Ngai, E.W.T. and Chan, E.W.C. (2005) ‘Evaluation of knowledge management systems using
AHP’, Expert Systems with Applications, Vol. 29, No. 4, pp.889–899.
Percin, S. (2008) ‘Using the ANP approach in selecting and benchmarking ERP systems’,
Benchmarking: An International Journal, Vol. 15, No. 5, pp.630–649.
 A case study for project and portfolio management information system 381

PMI (1999) Project Management Software Survey, Project Management Institute, Newtown
Square.
PMI (2008) A Guide to the Project Management Body of Knowledge, Project Management
Institute, Newtown Square.
Raymond, L. and Bergeron, F. (2008) ‘Project management information systems: an empirical
study of their impact on project managers and project success’, International Journal of
Project Management, Vol. 26, No. 2, pp.213–220.
Rushinek, A. and Rushinek, S. (1991) ‘A product evaluation and selection system for project
management software’, Computers in Industry, Vol. 16, No. 3, pp.289–301.
Saaty, T. (1980) The Analytic Hierarchy Process, McGraw Hill, New York.
Saaty, T. (1989) ‘Group decision making and the AHP’, in Golden, B., Wasil, E. and Harker, P.
(Eds.): The Analytic Hierarchy Process: Applications and Studies, Springer-Verlag,
New York.
Saaty, T.L. (1996) Decision Making with Dependence and Feedback: The Analytic Network
Process, RWS Publications, Pittsburgh.
Tindale, R. (2003) ‘Group decision making’, in Hogg, M. and Cooper, J. (Eds.): Sage Handbook of
Social Psychology, Sage Publications, London.
Tyler, T. and Smith, H. (1998) ‘Social justice and social movements’, in Gilbert, D., Fiske, S. and
Lindsey, G. (Eds.): The Handbook of Social Psychology, McGraw Hill, Boston.
Vaidya, O.S. and Kumar, S. (2006) ‘Analytic hierarchy process: an overview of applications’,
European Journal of Operational Research, Vol. 169, No. 1, pp.1–29.
Voulgaridou, D., Kirytopoulos, K. and Leopoulos, V. (2009) ‘An analytic network process
approach for sales forecasting’, Operations Research: an International Journal, Vol. 9, No. 1,
pp.35–53.
Wang, R. (2007) Introducing Project-based Solutions – Enterprise Solutions for Project-based and
People-Centric Businesses, Forrester Research, Inc.
White, D. and Fortune, J. (2002) ‘Current practice in project management – an empirical study’,
International Journal of Project Management, Vol. 20, No. 1, pp.1–11.