A Study of Enterprise Resource Planning (ERP) System Performance

G Model
COMIND 2691 No. of Pages 13
Computers in Industry xxx (2015) xxxxxx
Contents lists available at ScienceDirect
Computers in Industry
journal homepage: www.elsevier.com/locate/compind
A study of enterprise resource planning (ERP) system performance

measurement using the quantitative balanced scorecard approach
Yung-Chi Shena,1, Pih-Shuw Chenb,* , Chun-Hsien Wanga,1
a
Department of BioBusiness Management, National Chiayi University, No. 580, Sinmin Road, Chiayi 600, Taiwan, ROC
b
Department of Business Administration, National Chiayi University, No. 580, Sinmin Road, Chiayi 600, Taiwan, ROC
A R T I C L E I N F O A B S T R A C T
Article history: Enterprise resource planning (ERP) systems have been used in integrating information and accelerating
Received 11 September 2014 its distribution across functions and departments with the aim to increase organizations operational
Received in revised form 11 May 2015 performance. Thus, it is worth measuring ERP system performance based on its impact to critical
Accepted 29 May 2015
performance of an organization: this requires a systematic method that bridges ERP performance
Available online xxx
measurement and key organizational performance. The hierarchical balanced scorecard (HBSC) model
with respect to multiple criteria decision-making is such a systematic approach to ERP performance
Keywords:
measurement. An ERP evaluation framework that integrates the balanced scorecard dimensions,
Enterprise resource planning
Balanced scorecard
linguistic variables, and non-additive fuzzy integral provides an objective approach to measuring both
Non-additive fuzzy integral the performance level of the ERP system and its contribution to the strategic objectives of high-tech
rms. Taking Taiwans high-tech rms as an example, this study demonstrates the effectiveness of this
integrated approach to measure the performance of ERP systems at the post-implementation stage under
evaluators subjective, uncertainty, and vagueness judgments.
2015 Elsevier B.V. All rights reserved.
1. Introduction vendors, such as SAP, Baan, Oracle, and People-Soft, provided

commercial software packages to favor the seamless integration of
The high-tech industry has played a central role in the economic all the information owing across a companys different functions.
development of Taiwan in recent decades. However, the character- Take SAPs R/3 package as an instance, it supports the integration of
istics of this industry include shorter product life cycles, increasing information ows across a companys functions, including nan-
global competitive pressure, and a variety of customer demands. In cials, human resources, operations and logistics, sales and
response to these trends, numerous high-tech rms in Taiwan have marketing. These commercial ERP packages promise an off-the-
accelerated the integration of production and manufacturing shelf solution to the problem of business integration for managers
information through the use of enterprise resource planning (ERP) who have struggled, at great expense and with great frustration, to
systems across the entire rm, in order to maintain a long-term coordinate incompatible information systems and inconsistent
competitive advantage. ERP is a tool to standardize and integrate operating practices [16]. For serving as the organizations platform
business processes to accelerate access to common resources to support such cross-functional integration, ERP architecture
across the organization so that ERP systems help organizations enables different business applications to share a common
facilitate information sharing and improve operational efciency database.
[8]. The successful deployment and use of ERP systems is critical to Various studies have asserted that ERP systems can increase
organizational performance and survival [42]. ERP is increasingly competitive advantage in the information technology (IT) era.
important in modern business because of its ability to integrate the Particularly in the e-business era, there has been a global trend to
ow of material, nance, and information to support organiza- integrate business processes based on the companys strategic
tional strategies [78,80]. Davenport [16] reported that many ERP implementation. A review of the potential benets from ERP
implementation can be classied into tangible and intangible
benets [1,67]. Tangible benets include reduction of inventory,
reduction of personnel, increased productivity, improvements in
* Corresponding author. Tel.: +886 5 2732834; fax: +886 5 2732826. orders management, quicker closing of nancial cycles, increase of
E-mail addresses: syc@mail.ncyu.edu.tw (Y.-C. Shen), hideko@mail.ncyu.edu.tw
on-time deliveries, improved customer service, reduction in IT
(P.-S. Chen), chwang@mail.ncyu.edu.tw (C.-H. Wang).
1
Tel.: +886 5 2732877; fax: +886 5 2732874.
and procurement costs, improvement of cash ow management,
http://dx.doi.org/10.1016/j.compind.2015.05.006
0166-3615/ 2015 Elsevier B.V. All rights reserved.
Please cite this article in press as: Y.-C. Shen, et al., A study of enterprise resource planning (ERP) system performance measurement using the
quantitative balanced scorecard approach, Comput. Industry (2015), http://dx.doi.org/10.1016/j.compind.2015.05.006
G Model
2 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx
increase of revenue and prots, reduction in transportation and strategic decision of high-tech rms implementing ERP systems.
logistic costs, and reduction in the need for system maintenance. Using the balanced scorecard dimensions, high-tech rms can also
Intangible benets refer to the increased visibility of corporate assess their relative performance after ERP implementation, the
data, new or improved business processes, improved responsive- results of which can then be used to understand and monitor how
ness to customers, improved communications, unanticipated performance affects strategic decision-making.
reduction in costs, better integration between systems, standardi- This paper is now organized as follows: Section 2 reviews the
zation of computing platforms, increased exibility, global sharing literature on ERP performance; Section 3 derives the ERP
of information, improved business performance, and improved performance criteria to respond to BSC perspectives; Section 4
visibility into the SCM process [1]. Despite these signicant constructs the hierarchical balanced scorecard framework for ERP
benets, one major problem is that ERP systems cannot measure performance measurement; Section 5 elaborates on the linguistic
the performance which the systems impact on the rms. As variables, fuzzy measures, and non-additive integral employed in
Davenport [16] pointed out, many companies failed to consider this study; Section 6 uses the proposed method to assess the ERP
whether the system they were evaluating will match their overall performance of high-tech rms; and nally, the results and
business strategy. The balanced scorecard (BSC) approach, though, conclusions are presented in Section 7.
is designed to support a variety of organizational performance
measurement structures. The original BSC, proposed by Kaplan and 2. Review of the relevant ERP and performance literature
Norton [33], was a performance measurement system consisting of
four dimensions: nancial, customer, internal business processes, 2.1. The implementation of ERP
and innovation and learning. It became a popular performance
assessment technique because it was not only easy to implement Various studies have been conducted to identify critical factors
across different departments but also provided a well-dened affecting the successful implementation of ERP. Many focus on
framework through integrating the tangible and intangible individual case studies and industry surveys, and have covered a
perspectives and delivering the rms objectives, and therefore broad range of research issues. Motwani et al. [46] applied a case
giving the business a competitive advantage. Chand et al. [8] thus study methodology to compare a successful and unsuccessful ERP
argued that the BSC approach may be an appropriate technique for implementation, nding that the main factors behind a successful
evaluating the performance of ERP systems, if well-dened implementation consisted of cautious, evolutionary, bureaucratic
performance measurement and related indicators are used to implementations backed by careful change management, network
keep the strategic targets on track. relationships, and cultural readiness. Yen and Sheu [79] also used
In theory, it is difcult to quantify information systems (IS) due case study method involving direct observation and systematic
to the intangible nature of many of the benets, such as improved interviews to examine ve US and Taiwanese manufacturing rms,
customer satisfaction. By applying the fuzzy set theory, this the results of which indicated that ERP implementation should be
research undertakes an empirical study of numerous high-tech aligned with a rms competitive strategy. Meanwhile, Ash and
rms with the aim of systematically measuring ERP system Burn [2] expressed their concerns with ERP implementation by
performance via decision-makers evaluations, taking the intan- using embedded, multiple case studies to investigate the complex
gibles and tangibles into account. Furthermore, the BSC proposed phenomenon of an e-ERP project. Umble et al. [67] pointed out a
by Kaplan and Norton [33] argues that the interrelationships exist new set of key issues for successful ERP implementation, which
among nancial, customer, internal business process, and learning was also considered by Sun et al. [63]. They studied ERP
and growth perspectives. Additionally, there are many situations implementation in 26 rms and used ve critical success factors,
where observations cannot be described accurately as, for instance, with a total of 22 attributes, in a simulation model to assess
when they depend on environmental conditions or on individual strategic ERP implementation. Mabert et al. [41] surveyed
responses [68]. Specially, the BSC framework employed in this 193 manufacturing rms in the US and pointed out that successful
study relies on evaluators subjective judgments, the imprecise and ERP implementation depended on the organizations size, motiva-
vague nature embedded in human perception is inevitable. tion, implementation strategies, the modules and functionalities
Therefore, fuzzy linguistic variable scale proposed by Hersh and implemented, and operational benets. Ehie and Madsen [22] also
Caramazza [29] provides a simple and heuristic method to capture adopted an empirical study surveying 36 manufacturing compa-
the meaning of natural language. In addition, the traditional multi- nies in the Midwestern region of the US, identifying the critical
criteria approach assumes that decision factors are independent. In issues affecting ERP implementation as project management
other words, the aggregation of performance value is additive principles, human resource development, business process re-
based on the independent relationship among decision factors. engineering, cost/budget issues, IT infrastructure, consulting
However, the interdependence among various factors is common services, and top management support. Given such a range of
in real world. To reect this reality, fuzzy measures [72,47]or perspectives, though, it is unlikely that a consensus on a nal
more generally, non-additive set functionscan be used [73]. denition of ERP implementation can be achieved.
Therefore, a nonlinear integral i.e., Choquet integral, or so-called
non-additive fuzzy integral, is recommended in place of the 2.2. The benets of ERP
traditional weighted average methods [73].
Consequently, this study intends to construct a systematic Another research theme focuses on the details of implementing
performance measurement framework based on hierarchical ERP and their related success and benet. Chand et al. [8] pointed
balanced scorecard (HBSC) for ERP and adopts fuzzy linguistic out that implementing an ERP system not only increased customer
variables incorporating with non-additive fuzzy integral to deal satisfaction and reduced operational costs but also eventually
with the ambiguity and vagueness existing in evaluators subjec- resulted in increased prots and growth of an organization.
tive judgments in order to reect the subjectivity, uncertainty, and Furthermore, Davenport [17] and Markus et al. [42] indicated that
interaction embedded in the HBSC framework and the process of the benets of an ERP system were related to process productivity
performance evaluation. Specically, the ndings would enable improvement, reduced error, and timely availability of consistent
decision-makers and managers to better understand the perfor- information. They also suggested that ERP benets should be
mance of ERP implementation and, more generally, would measured from different perspectives. Trimmer et al. [64] reported
contribute to the understanding of the performance level and that support for the continuing use of critical success factors
G Model
Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 3
helped focus on the benets of ERP in rural healthcare. Al-Mashari increased productivity, and reduced nancial cycles. However, a
et al. [1], on the other hand, argued that ERP benets could be comprehensive and systematic approach to measure ERP system
realized when a close connection is established between the performance is necessary, since both its implementation and
implementation approach and business process performance performance measurement are typically complex and difcult to
measures. Shang and Seddon [56], in fact, proposed a comprehen- assess. To address this dilemma, this study develops a performance
sive framework for assessing ERP benets at ve dimensional measurement system for high-tech rms based on the balanced
levels: operational, managerial, strategic, IT infrastructural, and scorecard framework and a rigid mathematical approach.
organizational. Meanwhile, Murphy and Simon [48] examined the
intangible benets and demonstrated how they could be 2.4. The performance measurement of ERP systems
incorporated into traditional evaluation methods (e.g., cost-benet
analysis) of ERP implementation. Most of these studies conducted The business environment is characterized by a high level of
empirical surveys or case studies to examine the extent of, and the uncertainty, thus the process of ERP system assessment involves
different factors affecting, ERP implementation, rather than numerous problems [74]. Because ERP implementation is a
constructing multiple, comprehensive aspects to understanding complex and uncertain project, rms that are successful in its
the benets of ERP. implementation need to carefully examine all related factors to
increase operational efciency. Markus et al. [42] suggested that
2.3. The performance of ERP the ERP implementation was an enormously complex undertaking
for organizations. ERP systems can affect nearly every aspect of
Up to now, a quite few literatures focus on the post- organizational performance and function, and measures for the
implementation ERP system performance evaluation. Only some success of ERP systems should reect this fact. Indeed, Richardson
of articles literatures discussed the impact of ERP systems on the [19] argued that a one-dimensional approach fails to explain
productivity and performance. Dehning and Richardson [19] information technology (IT) performance. Hence, an ERP system
proposed a generic framework to guide the development of an evaluation framework must consider multiple aspects and criteria
ERP performance measurement model. Based on a comprehensive to facilitate performance measurement against a rms strategies.
review of nancial accounting and nance literature on IT Performance measurement in this context is therefore an
performance measurement, their framework classied these integrated holistic and systematic concept. It needs to embody
existing studies according to ve paths. Meanwhile, Hunton the whole organizational strategy and capture tangible and
et al. [32] examined the longitudinal impact of ERP adoption on intangible aspects, cover qualitative and quantitative criteria,
rm performance and found that return on assets, return on and include aspects of synergy through integrating useful
investment, and asset turnover signicantly improved over a 3- information.
year period. Matolcsy et al. [43] conducted an empirical study with Although ERP systems have been recognized as a useful
publicly available nancial accounting data to analyze ERP infrastructure for many businesses integrating functional infor-
adoption on overall rm and business process performance. They mation systems, as well as information ows and business
also found that the ERP adoption led to sustained operational processes, typical implementation is quite complex. Existing ERP
efciencies, increased protability, improvements in accounts systems cannot evaluate the performance of every process in an
receivable management, and improved overall liquidity. In organization. Thus, no single performance measurement approach
addition, Law and Ngai [38] conducted an empirical study to or tool can provide ERP system performance evaluation following
investigate the success of ERP adoption, the results of which implementation over a period of time. Constructing a systematic
indicated the positive associations between the perceptions of ERP and holistic performance framework to assess ERP implementation
success and improvement in business processes and perceived is therefore essential to subsequent decision-making. Evaluators
organizational performance. frequently adopt the common ERP evaluation criteria as perfor-
On the other hand, Klaus et al. [35] integrated historical mance indicators without developing any that include cross-
analysis, a meta-analysis of representative IS literature, and a function and cross-functional measurement. However, this study
survey of academic experts to investigate ERP implementation. addresses this issue by analyzing the results of a survey of rms
Their studies addressed the complex question of how to assess the that have implemented ERP systems. More specically, it adopts
organizational benets derived from an ERP system. Similarly, to the balanced scorecard (BSC) as an ERP performance measurement
examine how an ERP system is expected to affect a specic rms approach and proposes an accessible model for ERP performance
performance, the work of Poston and Grabski [50] indicated a measurement tools, whereby the criteria that inuence perfor-
signicant improvement in a rms performance after the mance, as well as the model itself, offer a methodology to assess
implementation of its ERP system. Hitt et al. [30] also claimed the performance of different ERP implementations. In this context,
that ERP deployment had a signicant, measurable effect on a the BSC and non-additive fuzzy set theory approaches have the
rms performance. Nicolaou et al. [49] compared nancial data potential to produce an innovative evaluation method for ERP
from rms adopting enterprise information systems with that performance, which solves the existing difculties. The literature
from a matched control group. The results of their univariate related to the performance evaluation of ERP is quite scarce. Most
analysis of performance across time periods demonstrated that focused mainly on case studies or adopted qualitative techniques
rms adopting enterprise systems had a signicantly higher to analyze the success or failure of ERP implementation, rather
differential performance in the second year after implementation than designing and developing systematic approaches to evaluate
than the control group. Finally, Wang et al. [71] examined 300 rms ERP performance. This research aims to address this gap by
from Taiwans top 500 largest corporations and found that a providing a current understanding of ERP implementation perfor-
cohesive ERP project team had a signicantly positive effect on mance measurement and the consequent strategic objectives. In
overall team performance in the context of ERP implementation. particular, this study seeks to understand the performance
Although the aforementioned literature reveals ambiguity in evaluation issues with ERP system implementation, by expanding
evaluating and predicting whether ERP implementation has a on the existing approaches to and literature on performance
signicant impact on a rms performance, these empirical studies measurement of and management approaches to ERP implemen-
do demonstrate that ERP systems do deliver improved rm tation in decision-making activities.
performance; for example, increased customer satisfaction,
G Model
3. Derivation of a balanced scorecard for ERP performance consider common measurement criteria, making judgments based
measurement on the comprehensive framework of ERP performance. The
primary objective of the BSC in this study, however, is to identify
Chand et al. [8] proposed an improved tool named the ERP and dene the core set of performance measures that will enable
scorecard, which integrates Kaplan and Nortons balanced managers to understand the current state of the ERP system in
scorecard with Zuboffs automate, informate, and transformate order to monitor its future performance. Moreover, ERP perfor-
goals for evaluating the performance of ERP systems. They mance measurement needs a comprehensive framework that
suggested that the BSC framework could be an appropriate systematically guides the identication and evaluation of ERP
approach for the performance measurement of ERP implementa- performance measures. Thus, an ERP balanced scorecard frame-
tions. However, the questions of how to measure the performance work in this study will guide the identication and evaluation of
of ERP and what kinds of methodology for appropriately measuring critical ERP performance measures. A well-dened framework also
post-implementation ERP performance. Specically, there is a lack helps decision-makers/managers to re-examine the goals of ERP in
of an analytic framework and a rigidity approach for ERP their organization.
performance measurement in the existing literature. The BSC Based on the above discussion, this study highlights the
infrastructure can serve as a guide to identifying measures for relevance of developing a systematic and holistic model, which
evaluating the performance and strategic objectives of ERP involves internal business processes, and organizational, nancial,
systems, though. Up to now, only Chand et al. [8] have attempted and non-nancial factors, to measure ERP performance in an
to employ this approach, through case studies, in assessing the organization. Therefore, all the dependent and interactive per-
specic strategic impacts of ERP systems. However, a well-dened spectives related to the process of ERP performance evaluation
framework and approach to building ERP balanced scorecards is must be considered in order to attain strategic success. These
needed, aligning ERP implementation and operation with strategic criteria are in accordance with most ERP systems and must be
objectives through a series of quantiable performance measure- integrated within a hierarchical balanced scorecard (HBSC)
ment indicators. framework for ERP performance evaluation. With this purpose,
Before ERP implementation, there is no way to directly measure this study develops such a hierarchical framework and combines
and monitor the exact performance level of ERP commercial precise quantitative approaches to explain, understand, and
packages following their implementation. In fact, the complexity of identify the direct and indirect contributions of ERP implementa-
ERP systems makes it difcult for a single aspect or indicator to tion. Once the ERP balanced scorecard framework has been
assess the multiple perspectives of ERP implementation perfor- dened, it will guide the identication of the required indicators
mance measurement. Dehning and Richardson [19] argued that a for ERP performance measurement.
one-dimensional approach failed to explain information technol- Keeney and Raiffa [34] proposed ve principles that must be
ogy (IT) performance. Consequently, a rm planning to implement followed when formulating criteria: (1) completeness, the criteria
an ERP system must employ multiple evaluation criteria from must cover all important aspects of the decision-making problem;
different performance perspectives. ERP system performance (2) operational, the criteria must be meaningful for decision-
measurement can thus be viewed as a multiple criteria deci- making analysis; (3) decomposable, the criteria can be broken
sion-making (MCDM) problem. Moreover, Farbey et al. [23] argued down from a higher to a lower hierarchy to simplify the evaluation;
that IT managers often used simplistic, estimative quantitative (4) nonredundant, there must be no double counting of criteria;
factors only, as they were often unable to capture many of the and (5) minimum size, the number of criteria should be as few as is
expected qualitative and intangible indicators. Remenyi and feasible. To ensure the validity and reliability of the BSC-based ERP
Sherwood-Smith [53] and Davern and Kauffman [18] presented performance measurement framework and its corresponding
a similar argument that information systems were not easily criteria, this study reviewed multiple sources of information.
assessed, especially when it involved intangibles. One way to First, the four dimensions of the evaluation criteria with which to
interpret both tangible and intangible factors affecting information assess ERP performance were extracted from the literature.
system performance, though, is to employ the balanced scorecard Second, six experts were consulted about the critical ERP
(BSC), within which are embedded relevant tangible and intangible performance concerns that needed to be addressed during the
performance indicators. Kaplan and Norton [33] emphasized that measurement of ERP performance. Particularly, the performance
the BSC was only a template and must be customized for the measurement of intangible benets of ERP systems needs to rely
specic elements of a sector, organization, or industry. The BSC on qualitative information yielded by experts, which is one of
provides the quantitative and qualitative basis to a complex common applications in the eld of decision making [11]. This in-
decision-making process in which high-tech rms need to depth interview with six experts ensured the performance
measure ERP performance against multiple criteria. The original indicators to meet the ve principles. Thus, the ideal experts have
BSC, developed for performance measurement by Kaplan and to be familiar with the benets of ERP systems, the implementa-
Norton [33], employed performance metrics from nancial, tion of ERP, ERP processes, and performance assessment. As
customer and internal business processes, and learning and mentioned in this manuscript, this study interviewed three
growth perspectives. Thus, it linked nancial and non-nancial academic experts in information management and three industry
perspectives into an integrated performance measurement sys- experts with over 6 years experience in the information sector of
tem, aligning organizational goals and other traditional functions high-tech industry. The academic experts who dedicated to the
with corporate strategy by means of leading indicators (perfor- eld of ERP helped assess the completeness, decomposability, and
mance driver-oriented indicators) and lagging indicators (out- nonredundancy of the performance indicators through their
come-based indicators) to monitor strategy implementation. domain knowledge. The information sector of high-tech compa-
Kaplan and Norton [83] indicated that the original BSC turned nies in Taiwan is actually responsible for implementing and
business strategies into measurable indicators. BSC provided a maintaining ERP as well as information systems. Therefore, the
series of performance measurement indicators that could be three industry experts who have adequate experience in ERP
utilized to guide strategic direction and objectives. implementation and processes helped assess the performance
ERP performance evaluation should also consider the direct indicators to meet completeness, operationality, and minimum
impact from factors such as organizational processes, customers size. Although the criteria varied, it converged on the four
perspectives, and information integration. Evaluators usually dimensions of the BSC-based ERP performance evaluation
G Model
framework (see Fig. 1). The framework constructed in this study improve business processes [44] and increase business
involves both the intangible and tangible criteria of ERP perfor- integration [3]. Due to the improved business integration
mance measurement. In addition, based on Kaplan and Norton beneted from ERP systems, the investment of a rm results in
[83], each evaluation category, consisting of 47 indicators, is higher returns [52]. Therefore, increased return on investment
appropriate for performance measurement. Based on these is regarded as a nancial indicator of ERP systems. In addition,
principles, this study developed a framework that provides a the improved business processes leads to improved overall
comprehensive and objective performance measurement mecha- productivity [59]. Additionally, one of the objectives of an ERP
nism. The four dimensions of the BSC-based ERP performance implementation may be to standardize processes and central-
measurement system are as follows: ize the control over information. To reach this objective, rms
have to standardize their IT systems, which may improve the
(1) Financial perspective: the goal of the ERP system here is to operational effectiveness and efciency and reduce operational
reduce costs and improve return on investment (ROI). Chand costs [1]. Thus, reduction of IT operational costs is considered a
et al. [8] suggested that the nancial performance indicators performance indicator in nancial perspective to evaluate the
include reduction of computer operating costs, reduction of performance of ERP systems. Moreover, the customer relation-
business operating and administrative expenses, and reduction ship management incorporated into ERP systems was evi-
in inventory costs and stock outs. Ranganathan and Brown [52] denced to increase market share [54] and sales growth rate
reported that ERP systems with greater functional scope or [27,54], thanks to the implementation of ERP leading to quick
greater physical scope result in positive and higher returns. reaction to market opportunities [1]. Therefore, increased
Because an ERP investment implies a rms commitment to market share and sales growth rate should be considered the
Increased return on investment (i11)
Reduction of IT operational costs (i12)
Financial Improved overall productivity (i13)
perspective (FP) Increased market share (i14)

Sales growth rate (i15)
Economic value added (i16)
ERP users satisfaction (i21)
Response time to ERP users complaints (i22)
Ease of learning and using (i23)
Post-implementation ERP Performance Evaluation System
Customer
Enhancing customer relationship management (i24)
perspective (CP)
Increasing customer retention and loyalty (i25)
Customer profitability (i26)
Customers satisfaction with products/services (i27)
The update speed of information system (i31)
Innovative information staff training materials (i32)
Innovation and Improved training process (i33)
learning (IL) The number of new system research and

development projects (i34)
The capability to deploy new IS functionality (i35)
Process innovation capability (i36)
Ability to adopt new processes (i37)

Automating cross-functional processes (i41)
Internal business Cross-functional integration ability (i42)
process (IBP) IT system availability and uptime (i43)
Improved standard procedures across different
locations (i44)
Improved operational efficiency (i45)
Ability to integrate information systems (i46)

Reduced duplication of input resources (i47)
Fig. 1. HBSC performance measurement system for post-implementation.
G Model
nancial performance indicators to measure the ERP systems. This section has introduced a BSC-based ERP performance
Finally, [77] reported a signicant relationship existed measurement framework based on a literature review and expert
between ERP implementation and economic value added, consensus for constructing an assessment model for post-
based on the antecedent empirical evidences that ERP systems implementation ERP performance evaluation. The next section
resulted in positive nancial performance of rms. presents the hierarchical balanced scorecard (HBSC) ERP perfor-
(2) Customer perspective: the main functions of ERP implemen- mance measurement system and the methodology for perfor-
tation are to meet customer needs more effectively and mance measurement.
eventually increase customer satisfaction. The successful
implementation of ERP systems relies on the dedication of 4. HBSC and ERP performance methodology
ultimate users in companies. Thus, ERP users satisfaction [75],
response time to ERP users complaints [75], and the ease of The hierarchical structure provides insight and landscape into
learning and using [45] should be included as the performance performance measurement system for a well-dened performance
indicators in customer perspective. In addition, since meeting metric through a number of indicators. In addition, the hierarchical
customer needs and satisfying customers are the main structure has many characteristics, including top management
functions of ERP implementation, enhancing customer rela- controlling the execution of performance, the scope and relation-
tionship management [54], customer retention and loyalty ship among different strategic objectives, and the framework
[54], customer protability [27], and customers satisfaction whereby the performance metric can be transformed into
with products and services [75] are considered the perfor- measurable criteria. Within this structure, every level has an
mance indicators in customer perspective. objective, for example, four strategic objectives are deployed the
(3) Innovation and learning perspective: the goal of the ERP overall performance evaluation of the ERP system. Therefore, a
system here is to give employees the capability to learn and use hierarchical structure is a necessity for ERP performance mea-
the system. Thus, the quality and effectiveness of the training surement.
programs affects employees ability to not only use the new The hierarchical structure adopted in this study to measure the
system but also modify process operations. Wu and Wang [75] post-implementation ERP performance in high-tech rms is
indicated that the successful implementation of ERP systems shown in Fig. 1. As mentioned, the four BSC-based dimensions
relied on user satisfaction. ERP vendors could provide timely were derived from a comprehensive literature review and
support to update the systems [7,57,20] and training [51] to consultation with experts, conrming the appropriateness of the
improve the user satisfaction. Hence, the update speed of the 27 critical criteria. All participants agreed that the evaluation
information [51], innovative information staff training materi- criteria should be comprehensive and relevant to assessing post-
als [51], and improved training process [51] could serve as implementation ERP performance. Furthermore, the ve principles
performance indicators in customer perspective. Additionally, for selecting criteria suggested by Keeney and Raiffa [34] and that
Srivardhana and Pawlowski [58] suggested that ERP systems for selecting performance indicators proposed by Kaplan and
had positive impacts on business process innovation. To enable Norton [33] were used to formulate the hierarchy for post-
business process innovation through ERP systems, companies implementation ERP performance measurement in high-tech
need to evaluate the number of new system research and rms. Consequently, this study constructed a HBSC with three
development projects offered by ERP suppliers or implemen- levels. The rst level is the goal level, the ultimate goal of the
tation teams [58], the ability of ERP suppliers or implementa- evaluation being to determine the overall performance of the post-
tion teams to deploy new information system functionality implementation ERP system. The second level is the strategic
[26], the process innovation capability of ERP systems [58], and objective, which includes nancial perspective, customer perspec-
the capability of ERP systems to adopt new innovation process tive, innovation and learning perspective, and internal business
[58]. Thus, this study views the number of new system research process. The third level is the criteria level, which includes
and development projects, the capability to deploy new IS 27 evaluation criteria. It should be noted that the inner dependence
functionality, process innovation capability, and the ability to among the criteria, suggested by the experts, exists within the four
adopt new innovation processes as performance indicators in perspectives.
innovation and learning perspective. A BSC-based ERP system with a well-dened performance
(4) Internal business process: one of the widely discussed issues is hierarchy and appropriate evaluation processes guarantees more
the need for an ERP process t for business and process correct results and facilitates faster assessments for decision-
changes. Improving access to information on ERP systems makers. Decisions based on correct performance information may
across different functions could make some strategic decisions lead to more efcient and effective management of rms and
more operationally and tactically effective. Davenport [17] also adjustments in strategic objectives.
suggested that ERP implementation is often performed
alongside business process re-engineering, and so improving 5. Methodology
business process efciency is one of the important tasks for an
ERP system. The internal business process covers a reduced The balanced scorecard provides an understanding of top
rate of input errors, increased efciency of business processes, managements strategy and translates this into a series of strategic
and improved decision support [8]. Therefore, automating objectives with operational measures at lower levels, such as
cross-functional processes [8], cross-functional integration measurable performance criteria. This study proposes a method of
ability [8,69], improved standard procedures across different ERP performance evaluation that draws on the knowledge and
locations [8], reduce input resources [8], ability to integrate experience of experts who understand how performance level
information systems [1], improved operational efciency [8,4], achieves the minimum requirements and how strategy is revised
and IT system availability and uptime [56] are regarded as in a management control setting, through complex performance
performance indicators in the perspective of internal business evaluation processes. However, these processes in different
process, based on the characteristics intrinsic in the intangible domains depend on experts personal experience, knowledge,
benets of ERP systems [1]. background, situations, and state of mind, and they may have only
a vague perception about the degree of preference for one option
over another, and so cannot apply a precise value to their
G Model
preferences. Chen et al. [11] indicated that the evaluation using multiple criteria. In other words, the fuzzy integral method allows
judgments from experts is one of common applications in the eld the modeling of interactions between criteria [25]. Applying
of decision making. On the other hand, conventional qualitative linguistic variables, the non-additive fuzzy integral approach
criteria are too complex, or too ill-dened, to be measured enables researchers to account for the vague and imprecise
numerically. Therefore, it is more appropriate to present their semantics and the interdependence among criteria, so reducing
preferences by means of linguistic variables rather than crisp ones the problems of subjective judgments involved in decision support.
[81,82]. Fuzzy logic can enhance the efciency of measurements Several studies have adopted this method in performance
based on approximate reasoning algorithms and vague informa- evaluations [14,40,70], and this study also attempts to apply the
tion. Therefore, considering the fuzziness in the evaluation data non-additive fuzzy integral and fuzzy linguistic variablesas
and performance measurement process, linguistic variables are described in the following subsectionsto assess different high-
used to determine the degree of importance for all criteria and tech rms post-implementation ERP performance, based on the
assess the performance values of rms. Fuzzy linguistic variables BSC perspectives.
enable the vagueness and uncertainty of both qualitative,
subjective, imprecise data and human cognitive analysis of the 5.1. Determining the degree of importance of criteria and the
criteria to be conducted from the evaluation process. Wang et al. performance values of rms
[84] also indicated that the linguistic variable composed by
triangular fuzzy numbers was able to reduce experts subjective In the performance evaluation process, strategic objectives are
judgments. Because the interval of triangular fuzzy numbers set according to the available data and information, which are
depends on experts professional ability to the extent that the vague, imprecise, and uncertain by nature. To address this issue,
surveyed experts can clearly distinguishes the difference of fuzzy linguistic variables are capable of handling the vagueness
decision criteria [84]. and uncertainty of data and information. Fuzzy linguistic variables
Furthermore, it is usually assumed that the criteria are represent those values that are presented as words and sentences
independent and additive in multi-criterion decision-making; in human or articial language. In this study, fuzzy numbers
hence, the weighted average method is often applied to aggregate represent both the decision-makers subjective assessment of the
the importance weight of those criteria [66]. Also, one may argue degree of importance of the criteria and the performance value of
that traditional multi-attribute decision-making (MADM) meth- the rms ERP according to those criteria. Herrera et al. [28] and
ods, such as analytic hierarchy process (AHP) and simple additive Chen et al. [10] suggested seven-point linguistic scales for
weighting (SAW), are able to evaluate the importance of criteria quantifying humans verbal judgments to achieve tractability,
and aggregate the overall importance. However, these assumptions robustness, and better rapport with reality. Wang et al. [70]
are not always true of many real world situations [66,76], as suggested that the interval of triangular fuzzy numbers should be
illustrated by the BSC proposed by Kaplan and Norton [33] that determined by the surveyed experts consensus. Based on this
emphasizes the cause-and-effect relationship among different manner, this study adopted a seven-point linguistic scale and then
performance indicators. Ignoring the effects of such interdepen- consulted the experts who are the same objects surveyed in this
dency could lead to assessment bias and ineffective decisions [37]. study to objectively determine the interval values. The seven
In other words, the fuzziness and vagueness embedded in human triangular fuzzy numbers were used to describe the linguistic
perceptions and the interdependence among decision factors performance terms, according to a type of conversion scale: very
coexist in the process of performance measurement. To cope with poor (VP), poor (P), median poor (MP), fair (F), median good (MG),
these challenges, some studies have adopted methods based on good (G), very good (VG); with the corresponding fuzzy numbers
BSC dimensions, which are able to address the ambiguities or as (0.0, 0.0, 0.1), (0.0, 0.15, 0.3), (0.2, 0.3, 0.4), (0.3, 0.5, 0.7), (0.6, 0.7,
reect the interdependent relationships. For example, Huang et al. 0.8), (0.7, 0.85, 1.0), (0.9, 1.0, 1.0), respectively (see Fig. 2).
[31] assessed the performance of knowledge management by During the overall ERP performance evaluation, evaluators
using the BSC-based analytic network process (ANP). Tseng [65] perceptions of the criteria vary in terms of the degree of their
integrated fuzzy set theory, ANP, and decision-making trial and importance and performance values, which are reected by the
evaluation laboratory (DEMATEL) to determine the relative weight values of the linguistic variables. Given that m represents the
of each BSC-based performance indicator. Bhattacharya et al. [6] evaluators, their evaluations of importance and performance are
applied fuzzy ANP to construct a BSC-based performance expressed as p ~i ; i 1; 2; ; m. One way to integrate the linguistic
measurement framework for the green supply chain. Shaik and variable expressing decision makers evaluations of importance is
Abdul-Kader [55] integrated BSC and DEMATEL into a comprehen- to apply the fuzzy arithmetic proposed by [21]. Thus, the
sive performance measurement model to assess reverse logistics aggregation of importance and performance values determined
enterprises, which demonstrated the causal-effect interactions by m decision makers is presented by three vertices of triangular
among decision factors. Nevertheless, these studies did not fuzzy numbers, as seen in Equation 1:
attempt to aggregate the performance information from different
alternatives for comparison. To take account of interdependence
and aggregate data from various decision factors, Sugeno [61]
therefore proposed the fuzzy integral method, which applied fuzzy
measures [5] to the problems of addressing the effects of
interdependency. The results of fuzzy measures can be further
applied to assess the performance of relevant alternatives [60],
providing an alternative computational scheme for aggregating
information [13]. Unlike the traditional, additive assessment
methods, which do not consider the effects of interdependency
between the criteria [37] and adhere to the notion that the
importance of two criteria in a probability framework is nothing
more than the sum of the importance of the individual criteria,
fuzzy measures are more exible in accepting that there are
greater or lower values than simply the sum of the importance of Fig. 2. Membership functions for seven performance values.
G Model

00X m 0X m 0X m 1
p p p This particular fuzzy measure is called the l-fuzzy measure,
BB pij B pij B pij C
BB p1
~ ij p j; M p ; R p BB
B p1
B
B p1
B
C
C because it has to satisfy l additively, and is also known as the
P L i ij ij BB m ; B m ; B m C (1)
Sugeno measure [61].
@@ @ @ A
Assuming that X fx1 ; x2 ; . . . ; xn g and PX is the power set of
X, the set function g: PX ! 0; 1 is a fuzzy measure, which is non-
~ ij p j;
where P M pij ; R pij are triangular fuzzy numbers repre- additive and preserves the following properties: 8A; B 2 bX,
L i
senting their points in the left, middle, and right positions, and L Pij , A \ B f, and gl A [ B gl A gl B lgl Agl B for
1 < l < 1. To differentiate this measure from other fuzzy
M Pij ,
and R Pij indicate the importance of dimension i and criterion j.
measures, gl denotes the l-fuzzy measure. When l 6 0, the
However, the fuzzy numbers for the importance weights of
l-fuzzy measure g is non-additive; otherwise, l 0 means that
decision factors need to be converted into crisp numbers to be of
the l-fuzzy measure g is additive and there is no interaction
subsequent use. Many defuzzication methods have been devel-
between decision factors [36]. Additionally, the l-fuzzy measure of
oped, including center of sum, center of gravity, mean of maxima,
the nite set can be derived from fuzzy densities, as indicated in
and a-cut. The defuzzication method proposed by Chen and Klein
Eq. (4) [34,39]:
[9], though, is a highly sensitive and effective approach, whereby
the precise importance value can be obtained and further X
n X
n1 X
n
incorporated into the fuzzy integral to aggregate the overall gl fx1 ; x2 ; . . . ; xn g gi l gi1 gi2
i1 i1 1i2 i1 1
performance of alternatives. Chen and Klein [9] employed a
~ ln1 g1 g2 ; . . . ; gn
method utilizing fuzzy subtraction of a referential rectangle (R;
1Y n
~ the
here, considered as a fuzzy number) from a fuzzy number (X); j 1 lgi 1j for 1 < l < 1 (4)
l i1
rectangle is derived from multiplying the height of the member-
ship function of X ~ by the distance between the two crisp
maximizing and minimizing barriers. Equation 2 shows how this Based on the boundary conditions in Eq. (4), gl X 1, l can be
fuzzy subtraction can be performed at level ai : determined via Eq. (5):
~a < > R
X ~ li ; ri c; d Y
n
i
li d; ri c; i 0; 1; 2; 1 (2)
l1 1 lgi (5)
i1
where < > and represent fuzzy subtraction and interval
subtraction operators, respectively; li and ri denote the left and
In the fuzzy measure space X; b; g, let h denote a measurable
~ and c and d, the left and right barriers. The
right loci of R, function from X to 0; 1. The fuzzy integral of h over A with respect
defuzzication rating of the fuzzy number can be obtained with to g is then dened as
Eq. (3): Z
X
n hxdg sup a ^ gA \ F a (6)
ri c A a20;1
~
DX i0
; n!1 (3) where F a fxjhx ag [73] and A represents the domain of a
X
n X
n
ri c li d fuzzy integral. When A X, the fuzzy integral can be presented as
R
i0 i0 hdg. For simplicity, consider the fuzzy measure g of X; PX
where n denotes the number of a-cuts; as n approaches 1, the sum where X is a nite set. Let h : X ! 0; 1 and assume, without loss of
X
n X
n generality, that the function hxi is monotonically decreasing in i,
is the measured area. In Eq. (3), ri c is positive, li d is for instance hx1 hx2 hxn . To ensure that the ele-
i0 i1 ments in X are renumbered, we use Eq. (7):
~ 1, if 0 x 1.
negative, and 0 DX Z
n
hxg _ hxi ^ gHi (7)
5.2. The l-fuzzy measure and non-additive fuzzy integral i1
where Hi fx1 ; x2 ; . . . ; xn g, and i 1; 2; . . . ; n. In practice, h can

An important aspect in building real applications for multiple be regarded as a given alternatives performance on a particular
criteria decision-making or performance measurement is the decision factor, and g represents the subjective degree of
interactive relationship among the feature criteria towards the importance for each decision factor. The non-additive fuzzy
strategic objective. In this study, such interaction exists in both the integral of hx with respect to g gives the overall assessment of
HBSC framework and the feature criteria. Fuzzy measures and the attribute. To simplify the calculation, the same fuzzy measure
fuzzy integrals are versatile operators that can be used in many of the Choquet integral is expressed as Eq. (8):
different domains for decision-making or evaluation, under the Z
assumption of interaction within the framework: criteria are not C hdg hxn gHn hxn1 hxn gHn1
always independent of each other in a real ERP performance hx1 hx2 gH1
evaluation process. Thus, fuzzy measures can be employed to hxn gHn gHn1
examine the interdependent relationships between the criteria hxn1 gHn1 gHn2 hx1 gH1 (8)
[12], which, along with the fuzzy integral, was the method
employed in this study to analyze the ERP performance and specify where 0 hx1 hx2 hxn 1, H1 fx1 g,
H2 fx1 ; x2 g, . . . , Hn fx1 ; x2 ; . . . ; xn g X. In the literature,
the interrelationship among the four dimensions of the HBSC R
framework. The fuzzy measure is used with the fuzzy integral to the fuzzy integral dened by hdg is termed a non-additive fuzzy
aggregate information, and can be dened as the subjective integral, and denotes the overall performance of the alternatives.
importance of a criterion during the evaluation process. Sugeno
and Terano [62] incorporated the l-additive axiom to simplify this
aggregation. In the fuzzy measure space X; b; g, let l 2 1; 1.
If A 2 b, B 2 b, A \ B f, then the fuzzy measure g is l-additive.
G Model
6. An empirical study: post-implementation ERP performance 6.2. Determining the degree of importance of performance indicators
measurement
In the non-additive fuzzy integral method, the weighting of
The optimality of the generated evaluation depends signi- decision factors starts from the bottom level of the decision
cantly upon domain experts and their professional knowledge and hierarchy. In the hierarchical structure, every item comprises an
experiences in adopting ERP systems. Generally, evaluation is a answer to a question and its associated degree of importance, and
procedure for determining the value of objects based on the values during the survey process, the importance of criteria is weighted
of their attributes. Evaluators assign linguistic values to fuzzy by using the fuzzy number. First, the three academic and three
variables in terms of given numerical (measurement) values. This industry experts as survey respondents evaluated the importance
section presents the data collection process and the results of the of the performance indicators on a questionnaire. Then these
empirical analysis. In addition, a worked example is provided in values were converted into triangular fuzzy numbers by using the
Appendix to illustrate the calculation process. linguistic variables shown in Fig. 2. The performance indicators
were subsequently weighted by aggregating the respondents
6.1. Data collection evaluation values with Eq. (1), which were nally entered into
Eq. (3) to calculate the degree of importance for each; the results
Researchers emailed the checklist of ERP performance meas- are shown in Table 1. As can be seen, the Innovation and Learning
ures to senior managers and senior auditors of high-tech rms to Perspective has the highest rank, with a weighting of 0.870,
determine the overall ERP implementation performance. These followed by Financial Perspective, 0.776, Customer Perspective,
were considered ideal respondents, because being responsible for 0.745, and lastly Internal Business Process, 0.531. Within each
a rms strategy, managers have a good understanding of the performance perspective, the weighting for each performance
internal/external operations of the business; while auditors are indicator similarly represents their degree of importance. The
responsible for monitoring whether the internal management highest ranking of 0.870 was shared between the three indicators
processes comply with required standards. Furthermore, as both IS of Update Speed of Information System, Innovative Information
managers and technology staff are responsible for ensuring data Staff Training Materials and Improved Training Process.
integrity and appropriate data-processing within ERP systems, the
HBSC is mainly designed to provide them with an overview for 6.3. Aggregating the performance values of high-tech rms ERP
evaluating organizational performance [24]. systems
The samples for this study were taken from those high-tech
rms that had implemented ERP systems over the last 3 years (i.e., Based on the importance weightings from the HBSC for post-
SAP, Oracle, Baan, JDEdwards, SSA, and so on). The hierarchical BSC implementation ERP systems, 72 senior IS managers and auditors
dimensions shown in Fig. 1 were then developed to measure each from 6 different high-tech rms evaluated the performance of
rms ERP performance and its impact on strategic objectives. As their ERP systems. As already described, the respondents
already mentioned, six senior managers and six auditorswith evaluation values were aggregated as fuzzy performance values
more than 10 years work experience on averageper rm were by using Equation 1 and converted into crisp numbers by using
asked to complete the survey from six different high-tech rms. Eq. (3). The l-value of each performance perspective was
calculated by using Eq. (5) with a corresponding density measure,
Table 1
Weighting of 4 performance perspectives and 27 performance indicators.
Perspective Weight (gi()) Indicator Weight (gi())

Financial perspective (FP) 0.776 Increased return on investment (i11) 0.620
Reduction of IT operational costs (i12) 0.647
Improved overall productivity (i13) 0.870
Increased market share (i14) 0.620
Sales growth rate (i15) 0.647
Economic value added (i16) 0.604
Customer perspective (CP) 0.745 ERP users satisfaction (i21) 0.800

Response time to ERP users complaints (i22) 0.776
Ease of learning and using (i23) 0.673
Enhancing customer relationship management (i24) 0.723
Increasing customer retention and loyalty (i25) 0.720
Customer protability (i26) 0.537
Customers satisfaction with products/services (i27) 0.694
Innovation and learning (IL) 0.870 Update speed of information system (i31) 0.870
Innovative information staff training materials (i32) 0.870
Improved training process (i33) 0.870
The number of new system research and development projects (i34) 0.750
The capability to deploy new IS functionality (i35) 0.844
Process innovation capability (i36) 0.783
Ability to adopt new processes (i37) 0.776
Internal business process (IBP) 0.531 Automating cross-functional processes (i41) 0.500
Cross-functional integration ability (i42) 0.540
IT system availability and uptime (i43) 0.647
Improved standard procedures across different locations (i44) 0.423
Improved operational efciency (i45) 0.540
Ability to integrate information systems (i46) 0.667
Reduced duplication of input resources (i47) 0.700
G Model
Table 2
Fuzzy measure and aggregated value for Company A.
R
Perspective Indicator h() gi() gl() c hdgl (l-value)
FP i11 0.769 0.620 gl(i12) 0.647 0.845 (0.999)

i12 0.909 0.647 gl(i12,i11) 0.866
i13 0.667 0.870 gl(i14, i11, i16) 0.948
i14 0.500 0.620 gl(i14, i11, i16, i13) 0.994
i15 0.500 0.647 gl(i14, i11, i16, i13, i12) 0.999
i16 0.667 0.604 gl(i14, i11, i15, i16, i12, i13) 1.000
CP i21 0.667 0.800 gl(i27) 0.694 0.761 (0.999)

i22 0.500 0.776 gl(i27, i24) 0.916
i23 0.667 0.673 gl(i27, i24, x26) 0.962
i24 0.769 0.723 gl(i27, i24, i26, i25) 0.990
i25 0.667 0.720 gl(i27, i24, i26, i25, i23) 0.997
i26 0.667 0.537 gl(i27, i24, i26, i25, i23, i21) 1.000
i27 0.769 0.694 gl(i27, i24, i26, i25, i23, i21, i22) 1.001
IL i31 0.667 0.870 gl(i36) 0.783 0.767 (0.999)

i32 0.769 0.870 gl(i36, i32) 0.972
i33 0.667 0.870 gl(i36, i32, i37) 0.995
x34 0.500 0.750 gl(i36, i32, i37, i35) 1.000
i35 0.667 0.844 gl(i36, i32, i37, i35, i33) 1.001
i36 0.769 0.783 gl(i36, i32, i37, i35, i33, i31) 1.001
i37 0.667 0.776 gl(i36, i32, i37, i35, i33, i31, i34) 1.001
IBP i41 0.500 0.500 gl(i47) 0.700 0.733 (0.998)

i42 0.500 0.540 gl(i47, i46) 0.901
i43 0.667 0.647 gl(i47, i46, i43) 0.966
i44 0.500 0.423 gl(i47, i46, i43, i45) 0.985
i45 0.500 0.540 gl(i47, i46, i43, i45, i44) 0.992
i46 0.667 0.667 gl(i47, i46, i43, i45, i44, i42) 0.998
i47 0.769 0.700 gl(i47, i46, i43, i45, i44, i42, i41) 1.000
gi, to represent the importance weighting of each indicator. Finally, Applying the same calculation process to the remaining ve
the Choquet integral value and l-value for the overall performance high-tech companies, their overall ERP system performance values
of one company was calculated by using Eq. (8). are shown in Table 4. The results show that Company E has the
Table 2 shows the fuzzy measure and aggregated value for one highest performance value (0.870), followed by Company B
of the six high-tech rms, Company Aa ctitious name used for (0.857), F (0.830), A (0.826), D (0.753), and C (0.675). According
condentiality reasons. The fourth column, gi(), in Table 2, repeats to the BSC dimensions therefore, Company E shows the best ERP
the importance weightings for performance indicators from the post-implementation performance.
fourth column in Table 1. The crisp performance value, h(), in the
third column in Table 2, shows Company As performance on each 7. Conclusion
indicator, while the l-value in the fth column presents the fuzzy
R
measure, and the Choquet integral value c hdgl in the last This study attempts to represent the performance of ERP
column represents the evaluators perception of the overall systems in the post-implementation stage as a measurement
performance. As can be seen, the Financial Perspective has the standard for high-tech rms. In this nal section, the value and
highest Choquet integral value of 0.845, which indicates that contribution of the BSC-based ERP performance measurement
Company A performs best in the Financial Perspective, according to system in this attempt will be summarized.
the judgment of the evaluators. Moreover, a l-value so close to First, this study develops an innovative approach by applying
1 means that there is complete dependence and mutual inuence the non-additive fuzzy integral to incorporate the BSC dimensions.
relationships among indicators. Its contribution lies in proposing a comprehensive ERP perfor-
The Choquet integral value of each performance perspective can mance measurement standard that takes account of individual
be further aggregated to determine the overall performance of performance indicators when analyzing the four BSC dimensions
Company A. Table 3 repeats Company As performance on each for each high-tech rm. This study then continues and extends the
perspective from the h() column and the importance weightings research on operational ERP performance measurement at rm
for each perspective from the gi() column in Table 2. By then level and develops an industry-specic algorithm to effectively
applying Eqs. (5) and (8), the fuzzy measure, l-value of gl , and measure the performance of ERP implementation.
R
the Choquet integral value, c hdgl , can be respectively Second, numerous factors that affect ERP performance are
calculated. The Choquet integral value of 0.826 represents the embedded in the balanced scorecard, which can thus increase both
overall performance of Company A. the precision of ERP performance measurement and the
Table 3
Fuzzy measure and overall performance value for Company A.
R
Company Perspective h() gi() gl() c hdgl (l-value)
A FP 0.845 0.776 gl(PFP) 0.776 0.826 (0.998)

CP 0.761 0.745 gl(PFP, PIL) 0.972
IL 0.767 0.870 gl(PFP, PIL, PCP) 0.994
IBP 0.733 0.531 gl(PFP, PIL, PCP, PIBP) 1.000
G Model
Table 4
Table 5
High-tech companies' ERP system performance values and rankings.
The triangular fuzzy number of respondents on i41.
Company Performance value Ranking
Respondent Linguistic performance Triangular fuzzy number
A 0.826 4
Expert A1 F (0.3, 0.5, 0.7)
B 0.857 2
Expert A2 F (0.3, 0.5, 0.7)
C 0.675 6
Expert A3 F (0.3, 0.5, 0.7)
D 0.753 5
Expert I1 MP (0.2, 0.3, 0.4)
E 0.870 1
Expert I2 MG (0.6, 0.7, 0.8)
F 0.830 3
Expert I3 F (0.3, 0.5, 0.7)
effectiveness of the subsequent decision-making on the successful 0:500

implementation of an ERP system. With the same procedure, the weights of four performance
Third, this study quanties ERP performance measurement in a perspectives and indicators can be derived and presented as gi() in
manner that is comparable across high-tech rms to improve the Table 1.
companies ERP performance against the benchmark in the In the fourth step, this study evaluated the performance of ERP
industry, and also extends the existing approaches to ERP systems from six different high-tech companies according to the
performance measurement and strategic decision-making by HBSC framework. Six senior managers and six auditors per
considering a series of critical factors. It therefore contributes to company were asked to evaluate their ERP systems. The perfor-
that body of studies attempting to investigate ERP system mance of ERP systems from six companies was evaluated on the
performance at rm level. performance indicators in HBSC with the same linguistic variable
In addition, the ERP performance measurement framework employed in step 1. Then, repeat the same step from 1 through 3 to
employed in this study provides a platform for further investiga- obtain the performance values on every indicators. The perfor-
tion into how different perspectives affect the performance of ERP mance values on the 27 indicators were shown as h() in Table 2.
systems, which is closely linked to strategic objectives. Finally, this In the fth step, the gi values of each sub-indicator in Internal
study presents an evaluation approach that can act as a reference Business Process were input into Eq. (5) to derive the l-value:
for better understanding the factors that affect ERP performance.
Moreover, one may argue that traditional MADM methods l 1 1 0:500l
could be applied under an interdependent, uncertain, and fuzzy 1 0:540l 1 0:647l 1 0:423l
environment. Thus, it is suggested to compare traditional MADM 1 0:540l 1 0:667l 1 0:700l
methods and non-additive fuzzy integral to clarify the impacts to
different results led by different basic assumptions.
This calculation was conducted by applying Mathematica 9.0.
Acknowledgments The l-value was 0.998 and presented in Table 2. The l-values of
other three dimensions were obtained with the same procedure.
This research is supported by the National Science Council The l-fuzzy measures were aggregated by using Eq. (4) in the
Taiwan, under grant NSC 102-2410-H-415-001-MY2. The insightful sixth step. To derive the l-fuzzy measure value for rm A, the
comments from reviewers are gratefully acknowledged. calculation started from the indicator with highest hi value in
Internal Business Process, i.e., Reduced Duplication of Input
Appendix Resources (i47), to the indicator with lowest hi value, i.e.,
Automating Cross-functional Processes (i41). The l-fuzzy measure
The calculation process was described as follows. First, the value for i47 was derived with Eq. (4) and shown as follows:
respondents evaluated the importance of performance indicators
gl i47 0:7
and the performance of their ERP systems according to the set of
indicators presented in the HBSC framework as presented in Fig. 1. Then, the indicator with highest hi (i47) was aggregated with the
Their evaluation values were converted into triangular fuzzy indicator with second highest hi, i.e., Ability to Integrate
numbers by using the linguistic variables shown in Fig. 2. Three Information Systems (i46):
academic experts in information management and three industry gl i47 ; i46 0:7 0:667 0:998 0:7 0:667 0:901
experts with over 6 years experience in the information sector of
high-tech industry were asked to evaluate the importance of The aggregated fuzzy integral value was then aggregated with the
performance indicators in the HBSC. Take the weighting of indicator with third highest hi, i.e., IT system availability and
Automating Cross-functional Processes (i41) for example, the six uptime (i43):
experts evaluation was presented as the following Table 5. gl i47 ; i46 ; i43 0:901 0:647 0:998 0:901 0:647
Second, the triangular fuzzy numbers of six respondents were 0:966
aggregated by using Eq. (1) as follows:
~ 41 0:3 0:3 0:3 0:2 0:6 0:3 ; 0:5 0:5 0:5 0:3 0:7 0:5 ; 0:7 0:7 0:7 0:4 0:8 0:7 0:333; 0:500; 0:667
P
6 6 6
Then, repeat the same procedure in this step until the indicator
Third, the aggregated triangular fuzzy number of i41 was defuzzied with lowest hi value.
by using Eq. (3) to obtain the weight of i41: Finally, the Choquet integral value was calculated by using
Eq. (8) to aggregate the overall performance of ERP systems from
~ 0:667 0 0:500 0 the bottom level of hierarchical framework for rm A. The
DX
0:667 0 0:500 0 0:333 1 0:500 1
G Model
calculation of Choquet integral value of International Business [29] H.M. Hersh, A. Caramazza, A fuzzy set approach to modiers and vagueness in
Process was shown as follows: natural language, J. Exp. Psychol. 105 (3) (1976) 254276.
[30] L.M. Hitt, D.J. Wu, X. Zhou, Investment in enterprise resource planning:
R business impact and productivity measures, J. Manag. Inf. Syst. 19 (1) (2002)
c hdg 0:5 1 0:5 0:5 0:998
7198.
0:5 0:5 0:992 0:5 0:5 0:985 0:667 0:5 [31] M.J. Huang, M.Y. Chen, K. Yieh, Comparing with your main competitor: the
single most important task of knowledge management performance
0:966 0:667 0:667 0:901 0:769 0:667
measurement, J. Inf. Sci. 33 (4) (2007) 416434.
0:7 [32] J.E. Hunton, B. Lippinott, J.L. Reck, Enterprise resource planning systems:
0:733 comparing rm performance of adopters and non-adopters, Int. J. Account. Inf.
Syst. 4 (3) (2003) 165184.
Repeat the same procedure in this step, the Choquet integral value [33] R.S. Kaplan, D.P. Norton, The balanced scorecard-measures that drive
of other three perspectives can be derived. With the same process performance, Harvard Business Rev. 70 (1992) 7179.
[34] R.L. Keeney, H. Raiffa, Decision With Multiple Objectives: Preferences and
from the rst through the last step, the Choquet integral value
Value Tradeoffs, John Wiley and Sons., New York, 1976.
which represents the overall ERP system performance of six [35] H. Klaus, M. Rodemenn, G.G. Gable, What is ERP? Inf. Syst. Front. 2 (2) (2000)
companies can be derived as presented in Table 4. 141162.
[36] G.J. Klir, Z. Wang, D. Harmanec, Constructing fuzzy measures in expert
systems, Fuzzy Sets Syst. 92 (2) (1997) 251264.
References [37] Y.C. Kuo, S.T. Lu, G.H. Tzeng, Y.C. Lin, Y.S. Huang, Using fuzzy integral approach
to enhance site selection assessmenta case study of the optoelectronics
[1] M. Al-Mashari, A. Al-Mudimigh, M. Zairi, Enterprise resource planning: a industry, Proc. Comput. Sci. 17 (2013) 306313.
taxonomy of critical factors, Eur. J. Oper. Research 146 (2) (2003) 352364. [38] C.C.H. Law, E.W.T. Ngai, ERP systems adoption: an exploratory study of the
[2] C.G. Ash, J.M. Burn, A strategic framework for the management of ERP enabled organizational factors and impacts of ERP success, Inf. Manag. 44 (2007) 418
e-business change, Eur. J. Oper. Res. 146 (2) (2003) 374387. 432.
[3] H. Barki, A. Pinsonneault, A model of organizational integration, [39] K. Leszczyn ski, P. Penczek, W. Grochulski, Sugenos fuzzy measure and fuzzy
implementation effort, and performance, Org. Sci. 16 (2) (2005) 165179. clustering, Fuzzy Sets Syst. 15 (2) (1985) 147158.
[4] H.M. Beheshti, C.M. Beheshti, Improving productivity and rm performance [40] W.T. Lin, S.C. Chen, H.F. Jang, H.H. Wu, Performance evaluation of introducing
with enterprise resource planning, Enterp. Inf. Syst. 4 (4) (2010) 445472. QS-9000 to the Taiwanese semiconductor industry, Int. J. Adv. Manuf. Technol.
[5] R.E. Bellman, L.A. Zadeh, Decision-making in a fuzzy environment, Manag. Sci. 27 (910) (2006) 10111020.
17 (4) (1970) 141164. [41] V.A. Mabert, A. Soni, M.A. Venkataramanan, The impact of organization size on
[6] A. Bhattacharya, P. Mohapatra, V. Kumar, P.K. Dey, M. Brady, M.K. Tiwari, S.S. enterprise resource planning (ERP) implementation in the US manufacturing
Nudurupati, Green supply chain performance measurement using fuzzy ANP- sector Omega, Int. J. Manag. Sci. 31 (3) (2003) 235246.
based balanced scorecard: a collaborative decision-making approach, Prod. [42] M.L. Markus, S. Axline, C. Petris, Learning from adopters experiences with ERP:
Plan. Control 25 (8) (2014) 698714. problems encountered and success achieved, J. Inf. Technol. 15 (4) (2000) 245
[7] P. Bingi, M. Sharma, J. Godla, Critical issues affecting an ERP implementation, 266.
Inf. Syst. Manag. 16 (3) (1999) 714. [43] Z. Matolcsy, P. Booth, B. Wieder The, economic benets of enterprise resource
[8] D. Chand, G. Hachey, J. Hunton, V. Owhoso, S. Vasudevan, A balanced scorecard planning systems: some empirical evidence, J. Account. Finance 45 (3) (2005)
based framework for assessing the strategic impacts of ERP systems, Comput. 439456.
Ind. 56 (6) (2005) 558572. [44] N. Melville, K. Kraemer, V. Gurbaxani, Information technology and
[9] C.B. Chen, C.M. Klein, An efcient approach to solving fuzzy MADM problems, organizational performance: an integrative model of IT business value, MIS
Fuzzy Sets Syst. 88 (1) (1997) 5167. Quart. 27 (2) (2004) 237263.
[10] C.T. Chen, C.T. Lin, S.F. Huang, A fuzzy approach for supplier evaluation and [45] M. Moalagh, A. Zare Ravasan, Developing a practical framework for assessing
selection in supply chain management, Int. J. Prod. Econ. 102 (2) (2006) 289 ERP post-implementation success using fuzzy analytic network process, Int. J.
301. Prod. Res. 51 (4) (2013) 12361257.
[11] S. Chen, J. Liu, H. Wang, Y. Xu, J.C. Augusto, A linguistic multi-criteria decision [46] J. Motwani, R. Subramanian, P. Gopalakrishna, Critical factors for successful
making approach based on logical reasoning, Inf. Sci. 258 (10) (2014) 266276. ERP implementation: Exploratory ndings from four case studies, Comput.
[12] T.Y. Chen, H.L. Chang, G.H. Tzeng, Using fuzzy measures and habitual domains Ind. 56 (6) (2005) 529544.
to analyze the public attitude and apply to the gas taxi policy, Eur. J. Oper. Res. [47] T. Murofushi, M. Sugeno, M. Macida, Non-monotonic fuzzy measures and the
137 (1) (2002) 145161. Choquet integral, Fuzzy Sets Syst. 64 (1) (1994) 7386.
[13] J.H. Chiang, Choquet fuzzy integral-based hierarchical networks for decision [48] K.E. Murphy, S.J. Simon, Intangible benets valuation in ERP projects, Inf. Syst.
analysis, IEEE Trans. Fuzzy Syst. 7 (1) (1999) 6371. J. 12 (4) (2002) 301320.
[14] H.K. Chiou, G.H. Tzeng, Fuzzy multiple-criteria decision-making approach for [49] A.I. Nicolaou, T. Stratopoulos, B. Dehning, Financial analysis of potential
industrial green engineering, Environ. Manag. 30 (6) (2002) 816830. benets from ERP systems adoption, J. Business Inf. Technol. 2 (1) (2003) 40
[16] T.H. Davenport, Putting the enterprise into the enterprise system, Harvard 50.
Business Rev. 76 (4) (1998) 121131. [50] R. Poston, S. Grabski, Financial impacts of enterprise resource planning
[17] T.H. Davenport, Mission Critical-realizing the Promise of Enterprise Systems, implementation, Int. J. Account. Inf. Syst. 2 (4) (2001) 271294.
Harvard Business School Press, Boston (MA), 2000. [51] J. Ram, D. Corkindale, M.L. Wu, Implementation critical success factors (CSFs)
[18] M.J. Davern, R.J. Kauffman, Discovering potential and realizing value from for ERP: do they contribute to implementation success and post-
information technology investments, J. Manag. Inf. Syst. 16 (4) (2000) 121143. implementation performance? Int. J. Prod. Econ. 144 (1) (2013) 157174.
[19] B. Dehning, D.J. Richardson, Returns of investments in information technology: [52] C. Ranganathan, C.V. Brown, ERP investments and market value of rms:
a research synthesis, J. Inf. Syst. 16 (1) (2002) 730. toward an understanding of inuential ERP project variables, Inf. Syst. Res. 17
[20] S. Dezdar, A. Sulaiman, Successful enterprise resource planning (2) (2006) 145161.
implementation: taxonomy of critical factors, Ind. Manag. Data Syst. 109 (8) [53] D. Remenyi, M. Sherwood-Smith, Maximize information systems value by
(2009) 10371052. continuous participative evaluation, Logist. Inf. Manag. 12 (1999) 1425.
[21] D. Dubois, H. Prade, Fuzzy Sets and Systems: Theory and Applications, [54] M.J. Schniederjans, Q. Cao, V.C. Gu, An operations management perspective on
Academic Press, New York, 1980. adopting customer-relations management (CRM) software, Int. J. Prod. Res. 50
[22] I.K. Ehie, M. Madsen, Identifying critical issues in enterprise resource planning (14) (2012) 39743987.
(ERP) implementation, Comput. Ind. 56 (2005) 545557. [55] M.N. Shaik, W. Abdul-Kader, Comprehensive performance measurement and
[23] B. Farbey, F. Land, D. Targett, Evaluating IT investments, J. Inf. Technol. (1992) causal-effect decision making model for reverse logistics enterprise, Comput.
71097122. Ind. Eng. 68 (2014) 87103.
[24] A.M. Ghalayini, J.S. Noble, T.C. Crowe, An integrated dynamic performance [56] S. Shang, B.P. Seddon, Assessing and managing the benets of enterprise
measurement system for improving manufacturing competitiveness, Int. J. systems: the business managers perspective, Inf. Syst. J. 12 (4) (2002) 271
Prod. Econ. 48 (3) (1997) 207225. 299.
[25] M. Grabisch, Fuzzy integral as a exible and interpretable tool of aggregation, [57] T.M. Somers, K.G. Nelson, A taxonomy of players and activities across the ERP
in: B. Bouchon-Meunier (Ed.), Aggregation and Fusion of Imperfect project life cycle, Inf. Manag. 41 (3) (2004) 257278.
Information, Physica Verla, Heidelberg, 1998. [58] T. Srivardhana, S.D. Pawlowski, ERP systems as an enabler of sustained
[26] T. Grbz, S.E. Alptekin, G.I. Alptekin, A hybrid MCDM methodology for ERP business process innovation: a knowledge-based view, J. Strategic Inf. Syst. 16
selection problem with interacting criteria, Decision Support Syst. 54 (1) (1) (2007) 5169.
(2012) 206214. [59] E. Stensrud, I. Myrtveit, Identifying high performance ERP projects, IEEE Trans.
[27] K.B. Hendricks, V.R. Singhal, J.K. Stratman, The impact of enterprise systems on Software Eng. 29 (5) (2003) 398416.
corporate performance: a study of ERP, SCM, and CRM system [60] M. Sugeno, S.H. Kwon, A clusterwise regression type model for subjective
implementations, J. Oper. Manag. 25 (1) (2007) 6582. evaluation, J. Jpn. Soc. Fuzzy Theory Syst. 7 (1995) 291310.
[28] F. Herrera, L. Martnez, P.J. Snchez, Managing non-homogeneous information [61] M. Sugeno, Theory of fuzzy integrals and its applications, Ph.d. Thesis, Tokyo
in group decision making, Eur. J. Oper. Res. 166 (1) (2005) 115132. Institute of Technology, Tokyo, 1974.
G Model
[62] M. Sugeno, T. Terano, A model of learning based on fuzzy information, [73] Z. Wang, K. Leung, J. Wang, A genetic algorithm for determining nonadditive
Kybernetes 6 (3) (1977) 157166. set functions in information fusion, Fuzzy Sets Syst. 102 (3) (1999) 463469.
[63] A.Y.T. Sun, A. Yazdani, J.D. Overend, Achievement assessment for enterprise [74] C.C. Wei, C.F. Chien, M.J. Wang, An AHP-based approach to ERP system
resource planning (ERP) system implementations based on critical success selection, Int. J. Prod. Econ. 96 (1) (2005) 4762.
factors (CSFs), Int. J. Prod. Econ. 98 (2005) 189203. [75] J.H. Wu, Y.M. Wang, Measuring ERP success: the ultimate users view, Int. J.
[64] L. Trimmer, L. Pumphrey, C. Wiggins, ERP implementation in rural health care, Oper. Prod. Manag. 26 (8) (2006) 882903.
J. Manag. Med. 16 (2/3) (2000) 113132. [76] J.L. Yang, H.N. Chiu, G.H. Tzeng, R.H. Yeh, Vendor selection by integrated fuzzy
[65] M.L. Tseng, Implementation and performance evaluation using the fuzzy MCDM techniques with independent and interdependent relationships, Inf.
network balanced scorecard, Comput. Educ. 55 (1) (2010) 188201. Sci. 178 (21) (2008) 41664183.
[66] G.H. Tzeng, Y.P. Yang Ou, C.T. Lin, C.B. Chen, Hierarchical MADM with fuzzy [77] L.J. Yao, S.G. Sutton, S.H. Chan, Wealth creation from information technology
integral for evaluating enterprise intranet web sites, Inf. Sci. 169 (34) (2005) investments using the EVA, J. Comput. Inf. Syst. 50 (2) (2009) 4248.
409426. [78] Y. Yao, H.C. He, Data Warehousing and The Internets Impact on Erp, March/
[67] E.J. Umble, R.R. Haft, M.M. Umble, Enterprise resource planning: April, IT Pro, 20003741.
implementation procedures and criteria success factors, Eur. J. Oper. Res. [79] H.R. Yen, C. Sheu, Aligning ERP implementation with competitive priorities of
146 (2) (2003) 241257. manufacturing rms: an exploratory study, Int. J. Prod. Econ. 92 (3) (2004)
[68] P.D. Urso, T. Gastaldi, An orderwise polynomial regression procedure for fuzzy 207220.
data, Fuzzy Sets Syst. 130 (1) (2002) 119. [80] Y. Yusuf, A. Gunasekaran, M.S. Abthorpe, Enterprise information systems
[69] S. Uwizeyemungu, L. Raymond, Impact of an ERP systems capabilities upon project implementation A Case Study Of ERP In Rolls-royce, J. Prod. Econ. 87 (3)
the relisation of its business value: a resource-based perspective, Inf. Technol. (2004) 251266.
Manag. 13 (2) (2012) 6990. [81] L.A. Zadeh, The concept of a linguistic variable and its application to
[70] C.H. Wang, I.Y. Lu, C.B. Chen, Integrating hierarchical balanced scorecard with approximate reasoningII, Inf. Sci. 8 (4) (1975) 301357.
non-additive fuzzy integral for evaluating high technology rm performance, [82] L.A. Zadeh, The concept of a linguistic variable and its application to
Int. J. Prod. Econ. 128 (1) (2010) 413426. approximate reasoningI, Inf. Sci. 8 (3) (1975) 199249.
[71] E. Wang, H.W. Chou, J. Jiang, The impacts of charismatic leadership style on [83] R.S. Kaplan, D.P. Norton, The Balanced Scorecard: Translating Strategy into
team cohesiveness and overall performance during ERP implementation, Int. J. Action, Harvard Business School Press, Boston (MA), 1996.
Project Manag. 23 (2005) 173180. [84] C.H. Wang, I.Y. Lu, C.B. Chen, Evaluating rm technological innovation
[72] Z. Wang, G.J. Klir, Fuzzy Measure Theory, New York, Plenum, 1992. capability under uncertainty, Technovation 28 (6) (2008) 349363.

A Study of Enterprise Resource Planning (ERP) System Performance

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

A Study of Enterprise Resource Planning (ERP) System Performance

Enviado por

Direitos autorais:

Formatos disponíveis

G Model

COMIND 2691 No. of Pages 13

Computers in Industry xxx (2015) xxxxxx

Contents lists available at ScienceDirect

A study of enterprise resource planning (ERP) system performance

1. Introduction vendors, such as SAP, Baan, Oracle, and People-Soft, provided

2 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 3

4 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 5

Increased return on investment (i11)

Reduction of IT operational costs (i12)

Financial Improved overall productivity (i13)

perspective (FP) Increased market share (i14)

Innovation and Improved training process (i33)

learning (IL) The number of new system research and

Ability to adopt new processes (i37)

Ability to integrate information systems (i46)

Fig. 1. HBSC performance measurement system for post-implementation.

6 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 7

8 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

where Hi fx1 ; x2 ; . . . ; xn g, and i 1; 2; . . . ; n. In practice, h can

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 9

Perspective Weight (gi()) Indicator Weight (gi())

Customer perspective (CP) 0.745 ERP users satisfaction (i21) 0.800

10 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

FP i11 0.769 0.620 gl(i12) 0.647 0.845 (0.999)

CP i21 0.667 0.800 gl(i27) 0.694 0.761 (0.999)

IL i31 0.667 0.870 gl(i36) 0.783 0.767 (0.999)

IBP i41 0.500 0.500 gl(i47) 0.700 0.733 (0.998)

A FP 0.845 0.776 gl(PFP) 0.776 0.826 (0.998)

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 11

effectiveness of the subsequent decision-making on the successful 0:500

12 Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx

Y.-C. Shen et al. / Computers in Industry xxx (2015) xxxxxx 13

Você também pode gostar