Proposed procedure for establishing Theoretical Reference -

application to Key Performance Indicators (KPIs)

Edilson Giffhorn
Universidade Federal de Santa Catarina – Depto. Engenharia de Produção
Rod. Amaro A. Vieira, 655. Ap. 106-E – Itacorubi – Florianópolis - SC – CEP 88034-101
edilson.giffhorn@gmail.com

Leonardo Ensslin
Universidade Federal de Santa Catarina – Depto. Engenharia de Produção - LabMCDA
Campus Universitário – Florianópolis – SC – CP. 476 – CEP 88010-970 – e-mail:
ensslin@deps.ufsc.br

Sandra Rolim Ensslin

Universidade Federal de Santa Catarina – Depto. Engenharia de Produção - LabMCDA
Campus Universitário – Florianópolis – SC – CP. 476 – CEP 88010-970 – e-mail:
sensslin@mbox1.ufsc.br

William Barbosa Vianna

Universidade Federal de Santa Catarina – Depto. Engenharia de Produção
R. Cônego Bernardo, 100. Ap. 202 – Trindade – Florianópolis - SC – CEP 88036-570
wpwilliam@hotmail.com

1. Introduction
According to Cauchick et al. (2008), methodological research in production engineering was
neglected for decades, with the exception of a few initiatives. Although this study represents
an evolution, particularly with the increase of qualified production, the speed in which this
has occurred is unsatisfactory.
These authors affirm that the need for methodological research advancements in the field of
production engineering in the country is notorious. This evolution is essential for an increase
in master’s and doctoral level work and, above all, qualified production, especially in high
impact national and international journals.
In this context, this article aims to provide a debate on the initial aspects of research
methodology, that is, the determination of Theoretical Reference and proposes a structured
process for this determination, illustrating the proposal regarding Key Performance Indicators
(KPI).
Concept and potentialities analysis and usage limits of classic bibliometry is based on the
election of a concept that is best applied to the matter in question, and identification of the
need for fitting and complementary qualitative modelling in the surveying of bibliometric data
when systematic search instruments are used, in view of better contextualized evidence and
refinement of the theoretical reference.
According to Le Boterf (2003), informational competence cannot be separated from the
respective individual, and relevance is based on the fact that there is no single way for
competency in relation to a problem or situation nor is there one correct observable behaviour
pattern. This cognition and structuring process of the information recovery system needs user

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
architecture in the search strategy, and should not be restricted to search techniques and tools,
notwithstanding, intervention in the entire informational search cycle.
In practice, initial results of this proposal provided such an extensive universe of publications
that the need for design feedback with new search criteria and subsequent accurate textual
analysis was revealed. This procedure avoided abstract discarding based solely on
mathematical criteria of a generic bibliometric search.
Constructs generated from the theoretical debate on the importance of qualitative-quantitative
association and empirical application to KPI resulted in a new two-phase research design.
The first phase selected knowledge areas and quality indicators of search data bases and used
them as keyword inputs and keyword association generating journal and article clusters that
were analyzed in the second phase.
The second phase used criteria of adherence and alignment to the research object, with input
based on critical analyses of selected articles to identify publications aligned with the research
subject.
Researcher intervention during feedback of the second design, from reading of the results
from the first design, titles, abstracts and associated keywords, favours elimination of bias,
slants and stop words, which are frequent words in a text that do not provide any relevant
information or contribution when observed within a context. This diminishes the number of
words to be analyzed and stored in a search base.
2. Determination of theoretical reference
The need for process structuring to determine theoretical reference of scientific work is
mainly derived from the increasing number of publications from a characterized environment
of information globalization, shortening of the technological innovation cycle and swiftness
of the process of generation, propagation and survival of knowledge.
More than a decade ago Kostoff (1996) had already identified a general total of five thousand
articles produced every work day around the world. A decade later, Santos (2003)
approximately pinpointed the same rate built into the data base from the Institute for Scientific
Information (ISI) alone, representing 1% of the global scientific publications volume.
In 1760, there were around ten scientific journals in the entire world. Between 1750 and 1950,
the number of publications increased ten times every 15 years. Currently, there is an estimated
total of 100 thousand journals around the world (OLIVEIRA, 2005).
The importance of periodic debriefing of “state of the art” validity of a research subject is
multiple and has gained increasing relevance in the area of Production Engineering,
emphasizing that debriefing: a) helps to detect theory and method stages; b) highlights study
object issues that outline the implied possibility of new research; c) reveals the extent in
which recent research is related to previous research; d) allows advancement through real
addition or enhancement of previous knowledge, or through overcoming previous conceptions
and approaches to problem loopholes.
One of the most well-known accepted and adopted methods to identify the current scientific
development stage referent to a specific subject is called bibliometry. In Brazil, the use of this
method was intensified after the 70s due to studies of the Instituto Brasileiro de Bibliografia e
Documentação – IBBD, presently the Instituto Brasileiro de Informação Científica e
Tecnológica – IBICT.
If a specific journal published scientific work quoted by others, the conclusion is that this
work, and the publishing journal, caused an impact in the scientific community. The more this

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
work is used as reference for others in a given timeframe, the higher the scientific impact of
this particular work.
However, isolated impact factors may not be synonymous of quality and may hinder the
evolution of certain research subjects. An example is local and regional issues developed in
peripheral or emerging countries, such as studies in Local Productive Arrangements (LPA),
Local Development and even Logistics and Transport in Brazil, among others. There is a
remote chance, or non-equitable, that this production gains international visibility and is
consequently cited and considered as being of impact.
For Gibbs (1995), the quasi invisibility of less developed nations in the international scientific
scenario may be more a reflection of economic issues and interest slants of scientific journals
rather than the real quality of research in these countries.
In this sense, the logic that prevails is that highly quoted journals are increasingly mentioned
and read, attracting more good authors, while journals outside the elite nucleus have
progressively difficult access to quotation and analysis indexes, with lesser reading and
quotation.
In summary, it seems there are some erroneous conceptions regarding science, knowledge,
scientific and technological development, propagation and transmission of knowledge,
concrete tasks for national science and, in particular, for the higher education institutions
within their different historical realities, and institutional development that can hinder
adequate determination of theoretical reference for research.
For these and other reasons the use of bibliometry should not occur without qualitative
refinement, which is many times overlooked or incorrectly evidenced regarding theoretical
reference presentation, and should be confined to succinct presentation of the adopted
theoretical reference without unnecessary contextualization or evidence of modelling criteria
and criteria selection.
3. Bibliometric analysis and its variations
In addition to bibliometry, “state of the art” studies are validated in other scientific
performance surveying variations and forms. These other analytic forms, such as
scientometry, infometry and webometry comprise unclear boundaries in relation to
bibliometry, which is presented as a central, basilar, and structured concept of these
quantitative techniques.
These techniques seek to extrapolate the use of quantitative analysis for different information
sources, formal or informal, and, recently, to the internet environment with quantitative
analysis technique application that opens the possibility of applying bibliometric techniques
to links (SANCHO, 1990; SPINAK, 1996; TAGUE-SUTCLIFFE, 1992).
According to Tague-Sutcliffe (1992) and Macias-Chapula (1998), bibliometry can be defined
as a study of quantitative aspects of production, propagation and use of recorded information
to develop mathematical standards and models, and measure such processes and their results
to determine forecasts and support decision making.
For Tarapanoff (1995), bibliometry is the study of quantitative aspects of production,
distribution and use of recorded information from mathematical models for the decision-
making process. Rostaing (1997) also defined bibliometry as the application of statistical or
mathematical methods on a set of bibliographic references.
According to Spinak (1996), bibliometry allows the conduction of studies of scientific and
technological organization from bibliographic references to identify the authors, their

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
relations and tendencies. The consideration of science and technology as systems that
generate information, knowledge and innovation that require input to function and produce
results can lead to the construction of scientific and technological production indicators
through measuring applied input and obtained results.
For Spinak (1998, p.142), bibliometry is a discipline with multidisciplinary reach that
analyses the most relevant aspects and objectives of the printed community; studies
organizations and their scientific and technological sectors from bibliographical sources and
patents to identify the authors, their relations and tendencies; produces quantitative studies of
published physical units, bibliographic units or their substitutes.
López-Piñero & Terrada (1992, p.67) warns that the indiscriminate use of bibliometric
indicators is erroneous, as lack of knowledge regarding the structure and dynamics of a
scientific community, communication processes, information and integration developed in the
core of research activity, is comparable with the use of blood cell count without biochemical
or immunological association to determine a state of health or illness.
In this way, the bibliometric concept of Spinak (1998) that observes a multidisciplinary scope
and relations and tendency analysis is considered as being the most extensive in regards to the
promotion of integration of qualitative elements to the theoretical reference determination
process, qualifying this concept to display better quality as the isolated or dissociated use of
quantitative search instruments can generate errors that restrict, rather than extend, the
developmental potential of research.
4. The importance of qualitative modelling and the role of researchers
Contextualization, conducted with the aid of qualitative methods, prevents theoretical
reference from being used merely as support and justification of research applications,
incurring the possibility of limited contribution to the development of the knowledge in
question for development of Brazilian research on the subject. In other words,
contextualization through qualitative methods aids in the preparation of theoretically coherent
work, with little possibility of development as theoretical reference was selected without a
structured process that justified this contextualization.
Some necessary concerns are not always evidenced in research that merely adopts
mathematics search instruments for the election of theoretical reference, without explaining
which merit or quality criteria is being used and why.
Consequently, prior to association with bibliometric mapping through mathematical search
instruments, there must be election and evidencing of search criteria and study
contextualization which is coherent with research objectives and the studied object. This
process should specify quality evaluation considerations of that which is elected and
discarded.
Qualitative modelling integration comprises different factors, including the notion that the
scientific community is not hegemonic, but conditioned to environmental variables that
should be observed, such as local and regional interests that could cause confusion between
intrinsic quality and national and international visibility of some work, or financial criteria
related to available public resources for use licensing of some data banks as opposed to
others, which could hinder access to quality work, among other factors.
When dealing specifically with the science of research in the area of production engineering,
it is important to emphasize that new and emerging roles of Production Engineering in
globalized economies seem laden with needs that can no longer waive a personalization
perspective regarding work context and scientific production (ENSSLIN & VIANNA, 2008).

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
This leads to the incorporation of the pertinence of affirmation of Le Boterf (2003, p.93), who
considered that researcher competence consists of knowing how to mobilize and combine
personal resources (qualitative dimension) and instrumental resources (quantitative
dimension) to solve a specific research situation.
Subsequently, input search and the need for accurate analysis of what should be read and
discarded after the use of bibliometric instruments are as critical as researcher ability to
situate his or her work.
This implies generalistic limits to problems and privileged subjectivity (inherent to the
subject, therefore, personalized) in the process of defining and specifying what is important
(objectives, criteria, attributes) and unimportant in a research problem. It should be
emphasized that the nature of production engineering problems is essentially qualitative
(ROY, 1994; ENSSLIN; DUTRA; ENSSLIN, 2000; VIANNA & ENSSLIN, 2008).
Methodological choice in research often implies a necessary risk that is inherent to scientific
development. Evidencing of values and interests and the justification of a particular
methodological choice allows extended reach of scientific theories when queries on
conditions and limits of validity are encouraged. (ENSSLIN & VIANNA, 2008).
Dudziak (2007) studies and records different information competency levels of the researcher:
a) basic level – organize and locate information, develop usage skills for informational and
technological tools, namely, obtain full command through digital literacy; b) secondary level
– use technology, seek information, locate, organize, transform into knowledge, think
systematically, namely, activate cognitive processes incorporating skills and knowledge
through reflection; and, c) complex level – build meaning from information, knowledge and
learning (informational, cognitive, attitudinal and valorative dimension), which is knowledge
added to the notion of value.
In this sense, competency is an attribute of the individual, which is only materialized when
put into action in a work context. The researcher may need to remodel data obtained in a first
search and reintroduce new input to elucidate and know the context, and subsequently adapt
surrounding contingencies to intervene and solve problems efficiently in accordance with
objectives, value and research criteria.
5. Proposal of a process for the determination of Theoretical Reference
In light of the foregoing, the proposal is to follow a basic model and a practical search
representation schedule of a second informational competency level in the search for
theoretical reference. The basic model is shown in Figure 1.

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
 Knowledge Areas Data Base Selection

Bibliometric Search Instruments

OUTPUT
INPUT
Number of
Key-words publications
Key-word Clusters of
Combinations journals with
relevant and
irrelevant
publications

INTERSUBJECTIVITY
(Advisor-advised party debate; research group)

Figure 1 – Research design model to establish the theoretical reference

Firstly, it should be noted that the model is based on intersubjectivity, mainly through
brainstorming or another interactive technique, with a advisor or research group, conduction
of critical analysis of the research subject and definition of keywords, sets or association that
can be used for research.
Researchers associate keywords subjectively through the observance of objectives, criteria
and values. Keywords can be associated for an improvement of relevance measuring and
thematic similarity before or after evaluation of overlap results between associated keywords.
Once research data bases and knowledge areas have been defined, a set of representative
research subject keywords is established, with a boolean combination adopted for search
systems of data base websites. Exclusion keywords and basic model feedback can be used for
higher refinement and accuracy.
After obtaining a result, titles of all articles, keywords and abstract are examined. In the case
of an article that is outside the interest area, the associated keyword that should be avoided is
identified.
Withal, if an article title is overtly misaligned with the subject, even without an exclusion
keyword, the title is removed for not contributing to the subject.
After this process, abstracts of selected articles are scanned to identify articles that contain
discard criteria or, as in the case of titles analysis, contain a description of extremely
particularized application, or that does not lead to the core of the research subject.
The basic model shown in Figure 1 can be reused and functions as a process that is fed back
until satisfactory refinement through alteration of input, namely inclusion or exclusion
keywords.

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
In this second stage, refinement is based on alignment and adherence criteria of articles in the
identified population targeted in the first stage. Content analysis (titles and abstract) is used to
conduct a new search with results that discard or include an article for full reading, as
subsequently shown in the application.
This analysis process is repeated until articles can no longer be removed due to misalignment
with the research subject. This process of analysing the entire picture and exclusion through
rationale favours the creation of unbiased reference that represents the research subject.
The search schedule of theoretical reference determination requires additional cognitive skills
of the researcher regarding search refinement. Organization of material eventually demands
full reading as the reading of titles and abstracts may not be sufficient for a number of
reasons.
Finally, the process leads to the construction of quantitative histograms by: a) data base; b)
journal; c) author; d) temporal distribution; and, e) reference count; which provide an
extensive view of obtained results.
6. Application of Proposal for Theoretical Reference Identification
This section shows the application of the process proposal for determination of Theoretical
Reference by identifying the manner in which different performance evaluation approaches
can pinpoint Performance Indicators.
Application of the proposed process was conducted in a study for personalized construction of
performance indicators that would evaluate organizational performance. Accordingly, the use
of the proposed research design model should result in the identification of representative
publications that associate the way in which different performance evaluation approaches
identify Indicators that measure objective properties.
In accordance to the model in Figure 2, the data base was defined intersubjectively and
restricted to data provided by the “Portal de Periódicos” of CAPES (Coordenação de
Aperfeiçoamento de Pessoal de Nível Superior), observing data base accessibility criteria via
portals that allow viewing of full texts and data base access permission for use of reference
management tools.

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
 Data base Inclusion and
selection Exclusion key
words
(1st phase) Data base

Title analysis
Unified through inclusion
library and exclusion
criteria
Data base General (2nd phase) Articles
article library Article list

Y Reading of
Articles Aligned?
abstracts
N

N
Reading of Y Theoretical
Contributes? fundamentation
articles

Figure 2 – Theoretical Reference search process

A total of 28 data bases were selected. Surveying for removal of data bases that link to
previously selected data bases led to a total of 15 data bases listed in Figure 3.

Figure 3 – Data bases

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
Subsequently, a set of keywords was formed, which represent the research subject and the
boolean combination to be used in search engines of data base web sites.

Figure 4 – Keyword sets and associations

The process shown in Figure 2 was used and reused with two inclusion and exclusion criteria,
respectively. Selection for consultation comprised data bases in the areas of Administration,
Human Science, Applied Human Science, Engineering and Multidiscipline (inclusion
criteria). Data bases related to sport, art, chemistry and biology were excluded (exclusion
criteria).
Search criteria were divided into two groups, inclusion criteria and exclusion criteria. Articles
that met the inclusion criteria are those that correspond to keyword association, without
containing keywords of the exclusion criteria group. Exclusion criteria corresponded to
articles on subjects in the areas of biological science, health, agriculture, sport, art, chemistry
and physics.
The query process of selected data bases with the use of keywords and inclusion and
exclusion criteria resulted in 8780 articles. These articles and the use of reference
management tools comprised the article libraries referent to each consulted data base.
The different libraries were integrated into a single library that, after elimination of
redundancy between two versions of the same article from different websites, comprised 6770
selected articles.
In accordance with the flux proposed in Figure 2, new inclusion and exclusion criteria were
selected. In this case, inclusion criteria considered the possibility that an article subject
provided theoretical contribution to the research subject through explicit quotation of
performance indicator use or form of obtainment.
Exclusion criteria comprised an approach based on particularized subjects, such as
applications in the areas of medicine and computing, or when the subject “performance
indicators” was absent from the title or abstract. This resulted in 228 articles for the reading
stage of respective abstracts.
The same inclusion criterion was adopted for the reading of 228 abstracts, resulting in 40
selected articles for the full text reading stage. After reading, five texts were discarded as they
originated from websites that required payment for access to the full text. A total of 35 articles
were subsequently selected with the contribution potential for research theoretical
fundamentation for obtainment process of key performance indicators, KPIs. Figure 5 shows
the number of articles obtained in each stage of the proposed schedule.

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
 Data base Inclusion and
selection Exclusion key
words
(1st phase) Data base

Title analysis
15 through inclusion
Unified
library and exclusion
criteria
8780 Data base General (2nd phase) Articles 228
article library Article list
articles articles
6770
articles

Y Reading of
40 articles Articles Aligned?
abstracts
N

N
Reading of Y Theoretical
Contributes? fundamentation
articles
35 articles

Figure 5 – Quantitative of articles obtained with application of the proposed process

7. Final Considerations
The proposal process proved efficient and effective regarding the purpose for which it was
designed and used. Inclusion and exclusion criteria were employed to survey published
articles between 1999 and 2008 that complied with both sets of research design criteria.
A total of 8780 articles were obtained and subsequently exported to reference management
software. However, one same article can be present during export of different data bases.
Therefore bibliometric differences of the reference manager were integrated with elimination
of redundancy, resulting in 6770 articles with contribution potential for theoretical research
validity.
The next stage in the proposed schedule was determination of inclusion and exclusion criteria
for title analysis of selected articles. Inclusion criteria determined that the article title should
quote the use or obtainment of Performance Indicators. Exclusion criteria was based on: (i)
absence of the terms Performance or Indicator in the title or article abstract; (ii) restricted
application of a specific context; (iii) description of applications in the area of health and
computing.
In this way, the sampling set with contribution potential for theoretical fundamentation was
reduced to 228 articles. The following stage comprised abstract reading with the same
inclusion and exclusion criteria of the previous stage. Reading of abstracts allowed the
elimination of articles that did not meet inclusion criteria, resulting in 40 articles with research
contribution potential.
The last stage of the schedule implied full reading of selected articles to identify those that
were aligned with research subject, and discard those which were misaligned. During full text
reading, 5 articles were discarded as payment was required for access to the full text. The

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia
remaining 35 articles were maintained and eventually comprised the theoretical
fundamentation of undergoing research on Performance Indicators.
References
CAUCHICK P. A. et al. Sessão Dirigida: Desafios para a Metodologia de Pesquisa em Engenharia da
Produção. Encontro Nacional de Engenharia de Produção (ENEGEP), Rio de Janeiro, 2008.
DUDZIAK, E. A. O bibliotecário como agente de transformação em uma sociedade complexa: integração entre
ciência, tecnologia, desenvolvimento e inclusão social. Ponto de Acesso. Revista do Instituto de Ciência da
Informação da UFBA, v. 1, n. 1, 2007.
ENSSLIN, L.; DUTRA, A.; ENSSLIN, S. MCDA: a constructivist approach to the management of human
resources at a governmental agency. International Transactions in Operational Research, n. 7, p. 79-100, 2000.
ENSSLIN L. & VIANNA, W. B. O Design na Pesquisa Quali-quantitativa em Engenharia de Produção –
Questões Epistemológicas. Revista Produção Online, Vol. 8, Num. 1, 2008.
GIBBS, W. W. Lost Science in the Third World. Scientific American, p. 76-83, August, 1995.
KOSTOFF, R. N. Performance measures for government-sponsored research: overview and background.
Scientometrics, 36(3), 281-292, 1996.
LE BOTERF, G. Desenvolvendo a competência dos profissionais. Porto Alegre: Artmed Editora, 2003.
LÓPEZ-PIÑERO, J. M. & TERRADA, M. L. Los indicadores bibliométricos y la bibliometría. Medicina
Clínica, vol. 98, nº. 2, p. 64-68, 1992.
MACIAS-CHAPULA, C.A. O papel da informetria e da cienciometria e sua perspectiva nacional e
internacional, Ciência da Informação, v. 27, n. 2, 1998.
OLIVEIRA, B. Produção cientifica nacional na ara de geociências: análise de critérios de editoração, difusão
e indexação em bases de dados. Ciência da Informação. v. 34, n. 2, p. 34-42. 2005.
ROSTAING, H. La bibliométrie et es techniques. Tolouse: Sciences de la Société, 1997.
ROY, B. On Operational Research and Decision Aid; EJOR 73, p. 23-26, 1994.
SANCHO, R. Indicadores bibliometricos utilizados en la evaluacion de la ciencia y la tecnologia: revision
bibliografica. Revista Española de Documentación Cientifica, v. 13, n. 3/4, p. 842-865, 1990.
SANTOS, R. N. M. Produção científica: por que medir? O que medir?. Revista Digital de Bibliotecomania e
Ciência da Informação. Campinas. V. 1, n. 1, p. 22-38, jul-dez. 2003.
SPINAK, E. Diccionario enciclopédico de bibliometría, cienciometría e informetría. Montevideo, 1996a. 245 p.
SPINAK, E. Los análisis cuantitativos de Ia literatura científica y su validez para juzgar Ia producción
latinoamericana. Boletín de Ia Oficina Sanitaria Panamericana, 120(2), 139-146, 1996b.
TARAPANOFF, K. Técnicas paratomada de decisão nos sistemas de informação. Brasília : Thesaurus, 1995.
TAGUE-SUTCLIFFE J. An introduction to infometrics. Information Processing & Management, v. 28, n. 1,p.
1-3, 1992.

ICIEOM International Conference on Industrial Engineering and operations Management, 06 - 09 October, 2009,
Salvador – Bahia