Evaluative Criteria for Qualitative

Research in Applied Linguistics:

Whose Criteria and Whose Research?
ANNE LAZARATON
Program in English as a Second Language
University of Minnesota
331C Nolte Center
Minneapolis, MN 55455
lazaratn@umn.edu
This paper examines various criteria that have been proposed for evaluating the increasing
number of empirical studies carried out using qualitative research methods, and it demon-
strates how such criteria may privilege certain forms of qualitative research while excluding
others. A broader disciplinary view is taken by defining qualitative research, and by discussing
in more detail the two qualitative traditions that have achieved prominence in applied linguis-
tics, ethnography, and conversation analysis. Then, select existing evaluative criteria for quali-
tative research proposed by applied linguists, as well as additional criteria developed outside
applied linguistics, are examined. Finally, the issue of criteriology is considered, on which
some of the assumptions underlying the existing evaluative criteria are based. To conclude,
this article discusses the complex relationship between research method and evaluative criteria
and the role of professional journals in establishing and validating such criteria.
QUALITATIVE RESEARCHHAS COME OF AGE
in applied linguistics, where it continues to flour-
ish. Inthe 1995 special topics issue of TESOL Quar-
terly, entitled Qualitative Research in ESOL, I
(Lazaraton, 1995) put forward a variety of evi-
dence to support the assertion that qualitative
research has made significant gains in terms of
visibility and credibility in recent years (p. 456).
More than 50 abstracts were submitted in re-
sponse to the original call for papers for that issue
of TESOL Quarterly; Qualitative Research Guide-
lines were set forth in each issue of TESOL Quar-
terly, and both qualitative and quantitative re-
search topics appeared in the Research Issues
column of the same journal. Although I con-
cluded that there seemed to be a lack of consensus
onthe definitions, principles, andvalue of qualita-
tive research, this was not necessarily problematic,
as long as the different methodological choices
were being presented in a balanced, objective way.
Five years later, my optimism on the status of
qualitative research in applied linguistics had
been tempered somewhat, based on my analysis
of 332 data-based articles in Language Learning,
Modern Language Journal, Studies in Second Lan-
guage Acquisition, and TESOL Quarterly from 1991
1997 (Lazaraton, 2000). Only 10% of the empiri-
cal articles were qualitative in nature, with TESOL
Quarterly publishing the majority of these inter-
pretive studies.
Nevertheless, there is still reason to be hopeful.
An invited colloquium on Setting Standards for
Qualitative Research in Applied Linguistics at the
2001 American Association for Applied Linguis-
tics (AAAL) conference in St. Louis was attended
by more than 100 conference participants, many
of whom engaged in a lively debate on the topic
in the question-answer session following the three
presentations. My own thinking on the topic of
research standards has itself been evolving over
the last 15 years, since the writing and publication
of The Research Manual: Design and Statistics for
Applied Linguistics (Hatch & Lazaraton, 1991), a
text that devoted a great deal of attention to the
evaluation of statistical research in applied lin-
guistics. In retrospect, this evaluation was a fairly
The Modern Language Journal, 87, i, (2003)
0026-7902/02/112 $1.50/0
2003 The Modern Language Journal
easy task, because there were numerous books on
quantitative research methods in the social sci-
ences espousing many of the same views on what
constitutes good quantitative or statistical re-
search. That is, one can locate various sets of
evaluative criteria for statistical research, criteria
that are meant to set or enforce some sort of
standard(s) of practice: The criteria are opera-
tionalizations of the standard(s).
1
But what of qualitative research? How can we
assess the quality of the increasing number of
studies now being carried out using qualitative
techniques? What role do evaluative criteria play
in the research process? Can any one set of crite-
ria be used to judge any or all forms of qualitative
research? My thesis, in answer to this last ques-
tion, is that they cannot: Existing criteria are in-
adequate for evaluating even the two forms of
qualitative researchethnography and conver-
sation analysisin which applied linguists cur-
rently engage. We need to remember that a set of
values is entailed by evaluative criteria for any
form of research, and any such criteria can define
the kinds of research we engage in, how and
where the results are reported, and who good
researchers are. Therefore, a careful examina-
tion of the assumptions underlying existing re-
search criteria is critical if we are serious about
conducting, reporting, and promoting rigorous
quality research in any form. To this end, I first
take a broader disciplinary view by defining quali-
tative research and by discussing in more detail
the two qualitative traditions mentioned above:
ethnography and conversation analysis. Then, I
focus on some existing evaluative criteria for
qualitative research proposed by applied lin-
guists, as well as some additional criteria devel-
oped outside the field of applied linguistics. As a
means of pulling together the ideas put forward
in the paper, I take up the issue of the criteriology
upon which some of the assumptions underlying
the existing evaluative criteria are based. I con-
clude with some thoughts on our next steps in
this endeavor to understand the complex rela-
tionship between research method and evaluative
criteria and then to develop method-appropriate
criteria.
A DEFINITION OF QUALITATIVE RESEARCH
Although it is true that qualitative inquiry is
currently marked by some degree of fractiousness
among the anthropologists and sociologists who
engage in it, it is worth considering how scholars
who do qualitative research conceptualize their
work. This definition is from Denzin and Lincoln
(2000), which is, if not currently the most authori-
tative volume on qualitative research, certainly
the most comprehensive:
Qualitative research is a situated activity that locates
the observer in the world. It consists of a set of inter-
pretive, material practices that make the world vis-
ible. These practices transform the world. They turn
the world into a series of representations, including
field notes, interviews, conversations, photographs,
recordings, and memos to the self. At this level, quali-
tative research involves an interpretive, naturalistic
approach to the world. This means that qualitative
researchers study things in their natural settings, at-
tempting to make sense of, or to interpret, phenom-
ena in terms of the meanings people bring to them.
. . . Qualitative research involves the studied use and
collection of a variety of empirical materialscase
study; personal experience; introspection; life story;
interview; artifacts; cultural texts and productions;
observational, historical, interactional, and visual
textsthat describe routine and problematic mo-
ments and meanings in individuals lives. Accord-
ingly, qualitative researchers deploy a wide range of
interconnected interpretive practices, hoping always
to get a better fix on the subject matter at hand. It is
understood, however, that each practice makes the
world visible in a different way. Hence there is fre-
quently a commitment to using more than one inter-
pretive practice in any study. (pp. 34)
2
This rather lengthy quotation highlights some
qualitative research themes with which the ap-
plied linguist should be familiar: Such research is
a situatedactivity that is interpretive andnatu-
ralistic innature; its data sources are fieldnotes,
observations, conversations, and so on. Inter-
estingly, texts on research and research methods
in applied linguistics tend to contrast the charac-
teristics and practices of qualitative and quantita-
tive research with a list of binary distinctions (as in
Larsen-Freeman & Long, 1991; see also Johnson,
1992; Nunan, 1992); others make tertiary or even
four-way distinctions (Chaudron, 1988; Seliger &
Shohamy, 1989). So, for example, dichotomies
suchas valid versus reliable, subjective versus objective,
holistic versus particularistic, and observational ver-
sus experimental are used to contrast qualitative and
quantitative research, respectively. Perhaps this
tendency to define qualitative research, not in its
own right, but in contrast to quantitative research,
results fromthe training of many applied linguists
as (primarily) quantitative researchers. As a result,
it is difficult for many to give up the objective,
replicable, and generalizable findings that
such research ideally (although not necessarily)
affords, in favor of work that is subjective and
ungeneralizable.
A second aspect of qualitative research, one
2 The Modern Language Journal 87 (2003)
which is not readily apparent in applied linguis-
tics work, is that in other fields, qualitative re-
search has a clear sociopolitical agenda. Denzin
(2000) notes: The next moment in qualitative
inquiry will be one at which the practices of quali-
tative research finally move, without hesitation or
encumbrance from the personal to the political
(p. 261). Although there are certainly applied
linguists who have as their primary, if not sole,
concern the extent to which research is answer-
able to larger moral and political questions
(Pennycook, 1994, p. 692; see also Kumaravadi-
velu, 1999; Low, 1999; Norton, 1997; Price, 1999),
and there is a trend towards situating research in
a sociopolitical context, it is worth remembering
that the issue of the sociopolitical nature of re-
search is not unique to qualitative research. It is
unfortunate that these two issues have become
confounded in our field because so little (if any)
published quantitative research in applied lin-
guistics asks or answers questions of a sociopoliti-
cal nature. Clearly, there are applied linguistics
scholars who are interested in the sociopoliltical
ramifications of what we do, whether that be in
teaching, in testing, or in research. However,
there is still a great deal of research being con-
ducted in applied linguistics that is agnostic on
these larger moral and political questions. Hav-
ing a sociopolitical agenda is not a requirement
for doing qualitative research, nor does having
one preclude investigating such issues quantita-
tively.
Finally, qualitative research is an umbrella term
for a very large group of research methodologies.
Two of these traditions,
3
conversation analysis
(CA) and ethnography, have become more preva-
lent in published applied linguistics research,
and as such merit a brief review. CA is a rigorous,
inductive approach to examining authentic spo-
ken discourse that has its roots in sociology, but
has recently been embraced by applied linguists
interested in validating oral language tests (Laz-
araton, 2002), in understanding second language
acquisition (SLA) (Markee, 2000), and in teach-
ing oral skills (Riggenbach, 1999). CA insists on
the analysis of real, recorded data, segmented
into turns of talk that are carefully transcribed.
Generally speaking, the conversation analyst does
not formulate research questions prior to analyz-
ing data; rather, questions emerge from the data.
The goal is to build a convincing and comprehen-
sive analysis of a single case, and then to search
for other similar cases in order to build a collec-
tion of cases that represent some interactional
phenomenon. Unlike other qualitative research
approaches such as ethnography, the conversa-
tion analyst places no a priori importance on the
sociological, demographic, or ethnographic de-
tails of the participants in the interaction or the
setting in which the interaction takes place.
Rather, the analyst, if interested in these issues,
attempts to detect their manifestations in the dis-
course as it is constructed, instead of assuming
some sort of omnirelevance beforehand. Fi-
nally, CA studies rarely report coding or counts of
data, because the emphasis in CA is on under-
standing single cases in and of themselves, not as
part of larger aggregates of data.
Ethnography is concerned with both descrip-
tions and interpretations of cultural behavior and
has its roots in anthropology. It entails a set of
techniques for data collection, data analysis, and
report writing for giving a comprehensive descrip-
tion of the people who make up a cultural unit
and the social practices in which they engage. Eth-
nography of communication emerged some 30
years ago from anthropological linguistics and is
associated with the work of Dell Hymes; it has
been embraced by applied linguists such as Davis
(1994); educational ethnography began even ear-
lier and has been primarily concerned with un-
derstanding the failure of certain groups in the
school context (e.g., Duff, 2002). Ethnographers
generally subscribe to a set of beliefs and prac-
tices, including the tenet that reality is both ho-
listic and socially constructed, and the goal of the
ethnographer is to uncover an emic (inside) per-
spective on the culture in question, even though
the researchers own positionality, or subjectivity, is
always present in the ethnographic endeavor.
Those who work in the ethnographic tradition
strive for prolonged engagement (perhaps a year or
more living or working in a community; Davis &
Henze, 1998); triangulation of data sources by em-
ploying multiple methods, researchers, and in-
formants; and a thick description of the cultural
context, through a detailed description of it. Fi-
nally, ethnographers engage in a cyclical process
of data analysis that results in grounded theory, and
they attempt to apply ethnographic findings to
real world problems.
The features of ethnography and conversation
analysis are summarized in Table 1.
Of course, Table 1 sketches out an incomplete
and deliberately contrastive heuristic; it is not in-
tended to, nor canit be, a definitive account of the
two research traditions. (But see the sources for
this table: Davis, 1995; Feldman, 1995; Gubrium&
Holstein, 2000; Have, 1999; Hopper, 1991; Mar-
kee, 2000; Miller, 1997; Moerman, 1988; Schiffrin,
1994; Silverman, 1998, 2000). In fact, some re-
searchers believe that the concerns of ethnogra-
Anne Lazaraton 3
phy and conversation analysis [are] connected in
intimacy (Hopper, 1991, p. 165), and that al-
though it is surely the case that some matters of
possible agreement within [conversationanalysis
ethnography of speaking] are sharply contested,
the[y] . . . nonetheless exhibit the suggestion of a
basic substructure through apparent confluences,
resonances, and comparabilities in conceptual
structures, persistent issues, and assumptions
(Wieder, 1998, p. 164).
In other words, as Markee (2000) points out,
Conversation analysis is epistemologically quite
close to ethnography as both these approaches
focus on the particular rather than the general
and also seek to develop a participants rather
than a researchers perspective on whatever phe-
nomena is being studied. Developing a parti-
cipants perspective involves developing a rich
description of context. However, conversation
analysts and ethnographers do not necessarily un-
derstand context in the same way (p. 26; see also
Fitch, 1998). Many of the differences evident in
Table 1 can be traced to these different interpre-
tations of context, and what counts in them.
In summary, this broad overview of qualitative
research ought to remind us, once again, that
qualitative inquiry is inherently valuable inits own
right, and not just a weak alternative to weight-
ier, more powerful quantitative research. Further-
more, it is likely that more qualitative researchers
in applied linguistics will want to consider more
seriously the sociopolitical aspects of their re-
search, because these issues are an important as-
pect of the qualitative research endeavor. Finally,
the breadth of qualitative research and the vari-
ability inherent in its many approaches must be
considered in developing, implementing, and
evaluating research standards. How well do ex-
isting criteria operationalize the salient features
of these two qualitative research approaches: con-
versation analysis and ethnography?
EVALUATIVE CRITERIA FOR QUALITATIVE
RESEARCH IN APPLIED LINGUISTICS
What sorts of evaluative criteria for qualitative
research have been proposed in the applied lin-
guistics literature? Probably the most widely
known criteria are the current Qualitative Re-
search Guidelines that appear in the Information
for Contributors section in each issue of TESOL
Quarterly (see Appendix A). Because of the edu-
cational role the Quarterly plays in modeling re-
search in the field (Teachers of English to
Speakers of Other Languages, Inc. [TESOL],
2001, p. 218), the guidelines are meant to en-
sure that . . . articles model rigorous qualitative
research (p. 219).
Now in light of the similarities and differences
evident in Table 1, neither the conversation ana-
lyst nor the ethnographer would have any argu-
ment with the data collection strategy of uncov-
ering . . . research participants perspectives,
because the primary goal of both approaches is
to model the demonstrated understandings of
participants (for CA, in spoken interaction only).
TABLE 1
Some Salient Characteristics of Ethnography and Conversation Analysis (CA)
Ethnography Conversation Analysis
Disciplinary Roots Anthropology Sociology
Analytic Goals What meaning is made How meaning is made
How people see things How people dothings
Context of Importance Cultural Sequential
Data Collection Procedures Informants Audio and/or videotapes of actual
Interviews interaction
Field work
Participant observation
Data Representation Thick description Detailed transcriptions using conventional
Field notes notation
Excerpts of interaction
Analytic Procedures Analytic induction Single & deviant case analysis
Triangulation
No Nos Impressionistic accounts Assigning a priori relevance to participant
Fixed category observations biographies or identities
Brief engagement Making claims based on speaker intention,
beliefs
Analyses of single turns
4 The Modern Language Journal 87 (2003)
The Guidelines then go on to discuss a number
of methodological concerns in Conducting the
Study, such as prolonged engagement, persis-
tent observation, triangulation, and thick de-
scription, data collection methods to which the
conversation analyst does not subscribe, at least
in the sense apparently intended in the Guide-
lines: The conversation analyst does persistently
engage with the recorded and transcribed data
for prolonged periods to produce a comprehen-
sive explanation for interactional phenomena,
but these are analytic canons, not data collection
concerns. Nor does the conversation analyst cre-
ate or test formal hypotheses or research ques-
tions, at least not in the ordinary positivistic sense
of these terms. Finally, because the locus of inter-
est for CA is the single case, comprehensive data
treatment in which data from all relevant sources
are analyzed is not an issue, because in CA only
data obtained from taped interactions and tran-
scribed according to conventional guidelines are
used. Even the Guidelines for reporting the data
suggest an ethnographic bias, in that contributors
are told that research questions should be clearly
stated and that a thick description of the context
must be provided. With respect to data analysis, it
is hard to envision how criteria that privilege ana-
lytic induction of data from several sources (as in
most ethnographic work) can be fairly applied to
conversation analytic studies that are based on
in-depth analyses of single cases taken only from
naturally occurring recorded interaction.
In other words, as I have argued elsewhere, the
Qualitative Research Guidelines now in TESOL
Quarterly . . . are clearly designed with ethno-
graphic research inmind andare difficult to apply
to many discourse-analytic studies . . . which use
data sources (especially carefully transcribed re-
corded interactions) that present the researcher
with a different set of data analysis and presenta-
tion concerns than does ethnography (Lazara-
ton, 1995, p. 461).
In fact, Qualitative Research Guidelines is a
misnomer; they should more accurately be
labeled Ethnographic Research Guidelines.
Where does this leave the discourse analyst, who
bases her analysis on a carefully transcribed inter-
action, but is unsure whether her study consti-
tutes good qualitative research? As it stands to-
day, these Guidelines reflect a rather narrow view
of qualitative research that, consciously or not,
privileges one research tradition over another.
More importantly, the Guidelines stress methodo-
logical rigor and yet are curiously silent on what
the larger evaluative concerns of qualitative re-
search should be.
A second set of Guidelines for Articles Report-
ing on Qualitative Research, which can be found
at the online journal Language Learning and Tech-
nology (LLT) Web site, appear in Table 2.
These guidelines are much broader than those
outlined by TESOL Quarterly; in fact, they are al-
most too broad, in that quantitative research
studies can, and often are, reported in just this
manner. Even though the LLT guidelines do not
appear to favor one sort of qualitative research
over another, it is hard to see how they are de-
signed specifically for investigations of a qualita-
tive nature.
Johnson (1992) presents a third set of guide-
lines (Table 3). These guidelines are labeled as
specific to ethnographic research, and, as such,
they are quite useful. Questions 9 and 10, which
are concerned with theoretical and pedagogical
implications of ethnographic work, are welcome
features of this list of evaluative criteria. Unfortu-
nately, Johnson did not propose guidelines for
discourse analytic research, although her Crite-
ria for Analyzing Case Studies appeared in a
chapter in which discourse is analyzed in some
exemplar studies. Her questions about field
techniques used in the research and cultural
interpretations made from it are not, unfortu-
nately, relevant to mainstream conversation ana-
lytic research.
Finally, although Markee might not have in-
tended the list in Table 4 to serve as a set of
TABLE 2
Language Learning and Technology Qualitative
Research Guidelines
The LLT editors recommend that authors of manu-
scripts based on qualitative research generally in-
clude the following sections in their articles:
Statement of the research question examined in
the study.
Description of the theoretical framework(s) under-
lying the research question.
Description of the methodological tradition in
which the study was conducted.
Relationship between the study and previous work
in the area under investigation.
Detailed description of the participants and re-
search site.
Detailed description of data collection and analysis
procedures.
Report of findings.
Limitations of the study.
Implication(s) of the study.
Note. From the Language Learning and Technology Web
site: http://llt.msu.edu/resguide.html
Anne Lazaraton 5
evaluative guidelines for CA, it does suggest how
we might proceed if we were to develop such
criteria. Although these methodological choices
do not preclude ethnographic research, their fo-
cus on transcribed communicative events as rele-
vant data makes a positive evaluation of research
that employs ethnographic data less likely than
the TESOL Quarterly or Johnson guidelines do.
Still, it is unclear if these characteristics of CA
could be operationalized into evaluative criteria
that go beyond dictums about research method-
ology. Points 2 and 3 in Table 4 are clearly state-
ments about methods, although the last two do
relate to outcomes of such research. Still, no
statement really encompasses the soundness of
the research itself.
It is worth noting that criteria established for
qualitative research outside of applied linguistics
are no more helpful in their inclusivity. For exam-
ple, Stewart (1998) proposes a checklist for eth-
nography (see Appendix B), based on criteria
that focus on how such research is conducted,
which implies that evaluating various charac-
teristics of methodological rigor is sufficient for en-
suring quality research. Her criteria include ve-
racity (i.e., validity), objectivity (i.e., reliability), and
perspicacity (i.e., generalizability). Another check-
list of sorts is proposed by Richardson (2000; see
Appendix C), who maintains that ethnography is
both a literary and a scientific endeavor and that
the consequences of such research must be con-
sidered. In both sets of guidelines, however, the
valid concerns that are addressed are not ones
that conversation analysts regard as central to
their work.
In sum, the four sets of guidelines are designed
to ensure sound research methodology and ap-
propriate research reporting in qualitative re-
search in applied linguistics. Though valuable for
delineating some of the broad parameters of
qualitative research, they fall short in capturing
some of the unique features of specific qualitative
research approaches. But perhaps we should step
back from the question of evaluating existing cri-
teria in order to consider how such criteria reflect
more fundamental, and often unexamined, be-
liefs about the research process itself.
CRITERIOLOGY IN QUALITATIVE
RESEARCH
How should we go about evaluating qualitative
research? I was surprised, but delighted, to learn
that there is a subarea of mainstream qualitative
research known as criteriology devoted to this topic.
Criteriology involves the constant search for per-
manent or universal criteria for judging qualita-
tive research (Garratt & Hodkinson, 1998, pp.
515516). Criteriologists suchas Garratt andHod-
kinson pose and consider two basic questions:
1. Should we be striving to establish agreed,
universal, preordained criteria, against which any
TABLE 3
Criteria for Assessing Ethnographic Reports
1. What are the goals of the ethnography? What is
the research problem?
2. In what contexts was the research conducted?
3. What is the group or case under study?
4. What conceptual and theoretical frameworks
inform the study?
5. What field techniques were used? For how much
time? In what contexts? What were the roles of
the ethnographer?
6. What analysis strategies were developed and
used? What levels and types of context were at-
tended to in interpretation?
7. What recurrent patterns are described?
8. What cultural interpretation is provided?
9. What is the contribution of the study to our
knowledge of sociocultural factors involved in
schooling in a second language and culture?
10. What are the stated implications for teaching?
Note. From Approaches to Research in Second Language
Learning (pp. 150151), by D.M. Johnson, 1992. Pub-
lished by Allyn and Bacon, Boston, MA. Copyright
1992 by Pearson Education. Reprinted by permission
of the publisher.
TABLE 4
CA-Oriented Methodology for a Social
Interactionist Approach to SLA Studies
1. Based on empirically motivated, emic accounts of
members interactional competence in different
speech exchange systems;
2. based on collections of relevant data that are ex-
cerpts of complete transcriptions of communica-
tive events;
3. capable of exploiting the analytical potential of
fine-grained transcripts;
4. capable of identifying both successful and unsuc-
cessful learning behaviors, at least in the short
term;
5. capable of showing how meaning is constructed
as a socially distributed phenomenon, thereby
critiquing and recasting cognitive notions of
comprehension and learning.
Note. From Conversation Analysis (p. 45), by N. Mar-
kee, 2000, Mahwah, NJ: Erlbaum. Reprinted with
permission.
6 The Modern Language Journal 87 (2003)
piece of qualitative research writing should be
judged?
2. If so, how do we choose which of the various
lists of criteria advanced in the literature we
should use in order to do that job?
Historically, criteriology can be traced along-
side the various moments of qualitative research
(Denzin & Lincoln, 2000) from the rejection of
positivist criteria for evaluating qualitative re-
search (i.e., reliability, validity, and generalizabil-
ity) to the substitution of interpretivist alterna-
tives, to the constructivist and relativist agendas,
and finally to the current political versions of
criteria that attempt to circumvent the problems
of relativism (see Seale, 1999, for more on this
issue).
According to Denzin and Lincoln (2000) and
Smith and Deemer (2000), there are three possi-
ble positions one can take on the issue of evalu-
ative criteria for qualitative research:
1. Foundationalism: There should be one set of
criteria for any kind of scientific research (i.e.,
positivism/rationalism: the criteria of reliability,
internal and external validity, and objectivity).
2. Quasi-foundationalism: There should be crite-
ria unique to qualitative research (i.e., postposi-
tivism, constructivism), which value (a) theory
generation, (b) empirical grounding and scien-
tific credibility, (c) generalizable/transferable
findings, and (d) internal reflexivity, in that re-
searcher, context, and strategy effects are taken
into account.
3. Nonfoundationalism: All such criteria should
be doubted and none should be privileged (i.e.,
postmodernism); altogether new criteria are
needed, in as much as judging inquiry is a practi-
cal, political, moral affair, not an epistemological
one (i.e., poststructuralism).
Now, it should come as no surprise that the ap-
plied linguistics literature on qualitative research
is scarce, and a discussion of criteriology in ap-
plied linguistics is nearly nonexistent. Clearly, ap-
plied linguists recognize the need for some sort
of criteria for evaluation, as Edge and Richards
(1998) observe by discussing this issue in terms of
standards:
Work is being referred to as ethnographic which does
not match up to the accepted standards of ethnogra-
phy. There seems to be a serious danger here that a
vicious circle may develop: The absence of an estab-
lished [ethnographic] tradition in TESOL leaves the
door open to poorly constructed naturalistic re-
search, and this provides ammunition for rationalists
who deny the value of such research. (p. 339)
But there is not necessarily agreement on what
the evaluative criteria should be. Davis (1995), in
the spirit of the current ethnographic tradition in
which she situates her own work, stresses not only
the requirements of qualitative research, but
also the consequences of qualitative research,
specifically the potential contributions such re-
search must make: To gain or maintain legiti-
macy within the applied linguistics field, not only
must qualitative studies meet the specific require-
ments of the approach used, but they must also
offer recognizable contributions to the field
(p. 436).
Perhaps the accessibility of qualitative research
in its written form may be a criterion we want to
consider. Lazaraton (2002) suggests that one of
the strengths of CA as a valid approach to lan-
guage testing research is just such accessibility
(although some scholars may argue that, for cer-
tain readers, numbers may seem more compre-
hensible than fragments of interaction).
Conversation analysis has much to recommend it as a
means of validating oral language tests. . . . Perhaps
the most important contribution that CA can make
. . . is in the accessibility of its data and the claims
based on them. That is, for many of us . . . highly
sophisticated statistical analyses . . . are comprehensi-
ble only to those versed in those analytic procedures.
. . . The results of CA are patently observable, even if
one does not agree with the conclusions at which an
analyst may arrive. As such, language testers who en-
gage in conversation analyses of test data have the
potential to reach a much larger, and less exclusive
readership. (p. 174)
4
Looking again at the existing evaluative criteria
for qualitative research in applied linguistics, the
Qualitative Research Guidelines from TESOL
Quarterly (Appendix A) imply a quasi-foundation-
alist perspectiveagain, the position that main-
tains that qualitative research requires unique
evaluative criteria, and not (necessarily) those
that are relevant for statistical researchwhich is
evident in the journals use of the terms cred-
ible and dependable. Dependability and trans-
ferability are the qualitative correlates of validity
and reliability (in the quantitative sense), respec-
tively (Davis, 1992).
The LLT guidelines appear to take a founda-
tionalist perspective on evaluating qualitative re-
search, in proposing criteria that are nearly iden-
tical to those for quantitative studies. As for the
criteria proposed by Johnson (Table 3) and Mar-
kee (Table 4), both appear to be quasi-founda-
tional in nature, mentioning particular terms or
research strategies that are part of the particular
research tradition. Johnson does bring up the
Anne Lazaraton 7
potential contribution of such research, and Mar-
kee suggests that CA work should be able, in
principle, to critique current SLA theories. Yet,
neither goes beyond quasi-foundational concerns
to any degree.
None of these guidelines prominently mention
nonfoundational issues; this seems to reinforce
the observation made earlier that qualitative re-
searchers in our field are not, at present, much
concerned with the sociopolitical issues in re-
search. Likewise, Stewarts (1998) checklist for
ethnography (Appendix B) is quasi-foundation-
alist, one that posits qualitative correlates for the
foundationalist criteria of validity, reliability,
and generalizability. Only Richardsons (2000)
ethnographic criteria (Appendix C), although
still quasi-foundational, incorporate some aspects
of nonfoundationalism, especially in terms of aes-
thetic merit or sociopolitical impact.
Noexamples of truly foundational checklists for
qualitative researchagain, the position that reli-
ability, validity, and generalizability are the criteria
of importance for any kind of empirical re-
searchcouldbe located, nor couldany nonfoun-
dational ones, suggesting that current qualitative
research orthodoxy seeks a middle ground on
this issue. Furthermore, althoughthere appears to
be neither agreement on which of these three po-
sitions (foundationalism, quasi-foundationalism,
and nonfoundationalism) one should espouse,
nor a definitive answer to the question of whether
these three positions are mutually exclusive, quali-
tative researchers appear to agree that criteriology
is a pressing issue for discussion, as Smith and
Deemer (2000) so eloquently note:
In the age of relativism the issue of who is making
judgments, about what inquiries, for what purposes,
and with whom one shares these judgments is of
critical importance. . . . As we approach judgment in
any given case, we have in mind or bring to the task a
list . . . of characteristics that we use to judge the
quality of that production . . . our lists are challenged,
changed, and modified not through abstracted dis-
cussions of the lists and items in and of themselves,
but in application to actual inquiries. That is, very
often something new comes along . . . the new does
not fit well with ones existing list of characteristics
. . . [this] immediately opens up the possibility that
one must reformulate ones list and possibly replace
the exemplars one calls upon in the never-ending
process of making judgments. However, the key here
is possibility, because inquirers may choose to pre-
serve their lists, as many often seem to do, and judge
the production as not even qualifying to be consid-
ered research. The latter is a comment, by the way,
that was often heard about qualitative inquiry in years
past. (pp. 887889)
5
Basically, Smith and Deemer remind us that, at
least in current thinking, truth is filtered
through the lenses of the judge, the context and
purpose of the inquiry, and the audience. What-
ever criteria we may believe we possess (con-
sciously or unconsciously) for judging research
quality, we needto understandboththeir relativity
and their positioned nature, and we must be open
to revising these criteria whenfacedwitha piece of
research that doesnt fit our existing lists.
In this section, I have suggested that main-
stream qualitative research is currently grappling
with the issues of criteria for evaluating research,
including whether we should attempt to establish
these criteria, and if so, what the criteria should
be. A review of recent literature points to a mid-
dle ground on this issue, where quasi-founda-
tional criteriathose that are unique to qualita-
tive researchare favored over both the
foundational (all research should adhere to the
same criteria) and the nonfoundational (all crite-
ria are problematic) extremes. It is evident that
applied linguistics has something to learn from
the criteriologiststhat is, agreement or dis-
agreement on evaluative criteria are based on
other underlying (and often unexamined) as-
sumptions about the research endeavor that must
be acknowledged and then argued. The exclu-
sionary power of any criteria would be a good
starting point for this long overdue discussion.
DISCUSSION
Three salient questions emerge from this re-
view:
1. Do we want to strive for one set of evaluative
criteria that apply to both ethnography and CA? If
so, what should those criteria be?
2. And, is there any way to get around the fact
that the criteria proposed are, for the most part,
really just statements about acceptable research
methods?
3. What is the role of the professional journals
in our field in this endeavor?
I hope to have shown that an affirmative answer
to the first question, even though perhaps desir-
able, is unlikely. Furthermore, it is worth asking if
the question itself is a version of foundationalism,
that we assume that there could, or should, be a
one-size-fits-all set of criteria. As for the criteria
themselves, the field must determine which are
most importantmethodological rigor? socio-
political impact? substantive contribution? report
accessibility? or other criteria altogether?
With respect to the second question, we should
8 The Modern Language Journal 87 (2003)
keep in mind what mainstream qualitative re-
searchers have readily acknowledged (Garratt &
Hodkinson, 1998):
All criteria for judging research quality contain
within them a defining view of what research is. It
follows that any attempt to preselect the criteria
against which a piece of research is to be judged is
also predetermining what the nature of that piece of
research should be. (p. 525)
That is, the nature of the research cannot be
separated from the methods used to carry it out,
which are implicated in the criteria used to judge
it (see also Rampton, Roberts, Leung, & Harris
[2002] on how theory-method relationships are
implicated in the accrual of disciplinary knowl-
edge).
As for this last question, it is true that TESOL
Quarterly to date has not published a CA study to
complement its growing list of published ethnog-
raphies; only in Applied Linguistics does conversa-
tion analytic research appear with any regularity
(e.g., Boyle, 2000; Mori, 2002; Wong, 2000). It
would be unfortunate if applied linguists who are
also conversation analysts continued to look to
other journals outside the field (e.g., Research on
Language and Social Interaction) as the only suit-
able publication outlets: Not only would the
larger applied linguistics readership be deprived
of learning about the outcomes of such studies,
but they would also fail to learn about the conver-
sation analytic technique itself.
Of greater concern, though, is the relative lack
of qualitative studies and criteria to evaluate them
in many of the journals in our field. Although
TESOL Quarterly has been singled out and taken
to task for its current Qualitative Research
Guidelines, it is still far ahead of the other jour-
nals in our field (e.g., Language Learning, Modern
Language Journal, and Studies in Second Language
Acquisition), which, to date, have published very
few qualitative studies at all,
6
much less set forth
criteria for evaluating such research. And even
though the existing TESOL Quarterly criteria are
ethnographically based, this is undoubtedly due
to the fact that CA had yet to take hold in applied
linguistics at the time the existing guidelines were
developed. In fact, a revision of the TESOL Quar-
terly Qualitative Research Guidelines is set to
debut in early 2003.
In a critical comparison of three classroom dis-
course studies which exemplify ethnography of
communication, CA, and functional systemic lin-
guistics, Green and Nixon (2002) highlight the
different expressive potential that these three
discourse analytic approaches offer. They note
that although each of the studies sets out a re-
search problem, highlights relevant phenomena,
and details how context shapes findings, none at-
tends to two remaining assumptions on which a
research tradition is based: What is considered a
valid account of some phenomenon, and what
standards of judgment should be used in evaluat-
ing these accounts? They conclude that there is
still a need for a broader discussion of these two
assumptions . . . within the critical discourse
among perspectives in applied linguistics (p.
404). We as applied linguists must begin the pro-
cess of considering, discussing, and arguing over
these issues, if our discipline is to continue to be
receptive to empirical research carried out using
qualitative techniques. I, for one, look forward to
taking part in this important endeavor.
ACKNOWLEDGMENTS
An earlier version of this paper was presented at the
Setting Standards for Qualitative Research in Applied Linguis-
tics invited colloquium at the 2001 conference of the
American Association for Applied Linguistics in St.
Louis, Missouri. I am grateful to Carol Chapelle, An-
drew Cohen, Kathryn Davis, Julian Edge, Keith Rich-
ards, Nick Saville, Elaine Tarone, Lynda Taylor, and to
the four anonymous reviewers for insights and sugges-
tions that greatly improved this paper. Any remaining
errors of omission or commission are mine alone.
NOTES
1
An anonymous reviewer pointed out, however, that
when we move beyond mainstream quantitative re-
search procedures, much less is agreed uponconfi-
dence intervals for effect size, for example.
2
From The Handbook of Qualitative Research, 2nd Edi-
tion (pp. 34), by N. K. Denzin and Y. S. Lincoln, 2000,
Thousand Oaks, CA: Sage. Copyright 2000 by Sage Pub-
lications, Inc. Reprinted with permission.
3
Obviously, it is somewhat arbitrary to choose these
two traditions to discuss in this paper; this approach may
strike some as exclusionary. However, the argument be-
ing put forward here is that the few guidelines for quali-
tative research that currently exist privilege one research
tradition to the exclusion of others; there are certainly
other qualitative research traditions that could be con-
trasted and that might also fail to cohere with these
existing guidelinesaction research, oral history, and
analysis of historical documents, to name a few.
4
From A Qualitative Approach to the Validation of Oral
Language Tests, by A. Lazaraton, 2002, Cambridge: Cam-
bridge University Press. Copyright 2002 by Cambridge
University Press. Reprinted with permission.
5
From The Handbook of Qualitative Research, 2nd Edi-
tion, (pp. 887889), by N. K. Denzin and Y. S. Lincoln,
Anne Lazaraton 9
2000, Thousand Oaks, CA: Sage. Copyright 2000 by
Sage Publications, Inc. Reprinted with permission.
6
Lazaraton(inpress) reports that out of 524empirical
articles published from 19912001 in these 4 journals,
74, or 14% were qualitative studies. The breakdown by
journal is as follows: Language Learning: 7 qualitative ar-
ticles out of 155 total, or 4%; Modern Language Journal:
23 qualitative articles out of 173, or 13%MLJ showed
the largest gain over the period, with two thirds of the
qualitative studies appearing in the last 4 years; Studies
in Second Language Acquisition: 5 articles with minimal
quantification out of 100 total, or 5%; TESOL Quarterly:
39 qualitative articles out of 96 total, or 41%.
REFERENCES
Boyle, R. (2000). Youve worked with Elizabeth Tay-
lor!: Phatic functions and implicit compliments.
Applied Linguistics, 21, 2646.
Chaudron, C. (1988). Second language classrooms. Cam-
bridge: Cambridge University Press.
Davis, K. A. (1992). Validity and reliability in qualitative
research on second language acquisition. TESOL
Quarterly, 26, 605608.
Davis, K. A. (1994). Language planning in multilingual
contexts: Policies, communities, and schools in Luxem-
bourg. Amsterdam: Benjamins.
Davis, K. A. (1995). Qualitative theory and methods in
applied linguistics research. TESOL Quarterly, 29,
427453.
Davis, K. A., & Henze, R. C. (1998). Applying ethno-
graphic perspectives to issues in cross-cultural
pragmatics. Journal of Pragmatics, 30, 399419.
Denzin, N. K. (2000). Aesthetics and the practices of
qualitative inquiry. Qualitative Inquiry, 6, 256265.
Denzin, N. K., & Lincoln, Y. S. (Eds.). (2000). The hand-
book of qualitative research (2nd ed.). Thousand
Oaks, CA: Sage.
Duff, P. A. (2002). The discursive co-construction of
knowledge, identity, and difference: An ethnogra-
phy of communication in the high school class-
room. Applied Linguistics, 23, 289322.
Edge, J., & Richards, K. (1998). May I see your warrant,
please?: Justifying outcomes in qualitative re-
search. Applied Linguistics, 19, 334356.
Feldman, M. S. (1995). Strategies for interpreting qualitative
data. Thousand Oaks, CA: Sage.
Fitch, K. L. (1998). Text and context: A problematic
distinction for ethnography. Research on Language
and Social Interaction, 31, 91107.
Garratt, D., & Hodkinson, P. (1998). Can there be crite-
ria for selecting research criteria?A hermeneu-
tical analysis of an inescapable dilemma. Qualita-
tive Inquiry, 4, 515539.
Green, J. L., & Nixon, C. N. (2002). Exploring differ-
ences in perspectives on microanalysis of class-
room discourse: Contributions and concerns. Ap-
plied Linguistics, 23, 393406.
Gubrium, J. F., & Holstein, J. A. (2000). Analyzing inter-
pretive practice. In N. K. Denzin & Y. S. Lincoln
(Eds.), The handbook of qualitative research (2nd ed.,
pp. 487508). Thousand Oaks, CA: Sage.
Hatch, E., & Lazaraton, A. (1991). The research manual:
Design and statistics for applied linguistics. Rowley,
MA: Newbury House.
Have, P. ten (1999). Doing conversation analysis: A practi-
cal guide. London: Sage.
Hopper, R. (1991). Ethnography and conversation
analysis after Talking Culture. Research on Language
and Social Interaction, 24, 161170.
Johnson, D. M. (1992). Approaches to research in second
language learning. New York: Longman.
Kumaravadivelu, B. (1999). Critical classroom discourse
analysis. TESOL Quarterly, 33, 453484.
Language Learning and Technology. (2002, May). LLT
research guidelines for quantitative and qualitative re-
search. Retrieved July 30, 2002, from the World
Wide Web: http://llt.msu.edu/resguide.html
Larsen-Freeman, D., & Long, M. H. (1991). An introduc-
tion to second language acquisition research. London:
Longman.
Lazaraton, A. (1995). Qualitative research in applied
linguistics: A progress report. TESOL Quarterly, 29,
455472.
Lazaraton, A. (2000). Current trends in research meth-
odology and statistics in applied linguistics. TESOL
Quarterly, 34, 175181.
Lazaraton, A. (2002). A qualitative approach to the valida-
tion of oral language tests. Cambridge: Cambridge
University Press.
Lazaraton, A. (in press). Quantitative research methods.
In E. Hinkel (Ed.), Handbook of research in second
language learning. Mahwah, NJ: Erlbaum.
Low, M. (1999). Exploring cross-cultural inscriptions
and difference: The effects of researchers posi-
tionalities on inquiry practices. TESOL Quarterly,
33, 292298.
Markee, N. (2000). Conversation analysis. Mahwah, NJ:
Lawrence Erlbaum.
Miller, G. (1997). Building bridges: The possibility of
analytic dialogue between ethnography, conversa-
tion analysis and Foucault. In D. Silverman (Ed.),
Qualitative research: Theory, method and practice (pp.
2444). London: Sage.
Moerman, M. (1988). Talking culture: Ethnography and
conversation analysis. Philadelphia: University of
Pennsylvania Press.
Mori, J. (2002). Task design, plan, and development of
talk-in-interaction: An analysis of a small group
activity in a Japanese language classroom. Applied
Linguistics, 23, 323347.
Norton, B. (1997). Language, identity, and the owner-
ship of English. TESOL Quarterly, 31, 409429.
Nunan, D. (1992). Research methods in language learning.
Cambridge: Cambridge University Press.
Pennycook, A. (1994). Critical pedagogical approaches
to research. TESOL Quarterly, 28, 690693.
Price, S. (1999). Critical discourse analysis: Discourse
acquisition and discourse practices. TESOL Quar-
terly, 33, 581595.
Rampton, B., Roberts, C., Leung, C., & Harris, R.
10 The Modern Language Journal 87 (2003)
(2002). Methodology in the analysis of classroom
discourse. Applied Linguistics, 23, 373392.
Richardson, L. (2000). Evaluating ethnography. Qualita-
tive Inquiry, 6, 253255.
Riggenbach, H. (1999). Discourse analysis in the language
classroom. Volume 1: The spoken language. Ann Arbor,
MI: University of Michigan Press.
Schiffrin, D. (1994). Approaches to discourse. Oxford:
Blackwell.
Seale, C. (1999). Quality in qualitative research. Qualita-
tive Inquiry, 5, 465478.
Seliger, H.W., & Shohamy, E. (1989). Second language
research methods. Oxford: Oxford University Press.
Silverman, D. (1998). Harvey Sacks: Social science &conver-
sation analysis. NewYork: OxfordUniversity Press.
Silverman, D. (2000). Analyzing talk and text. In N. K.
Denzin & Y. S. Lincoln (Eds.), The handbook of
qualitative research (2nd ed., pp. 821834). Thou-
sand Oaks, CA: Sage.
Smith, J. K., & Deemer, D. K. (2000). The problem of
criteria in the age of relativism. In N. K. Denzin &
Y. S. Lincoln (Eds.), The handbook of qualitative re-
search (2nd ed., pp. 877896). Thousand Oaks,
CA: Sage.
Stewart, A. (1998). The ethnographers method. Thousand
Oaks, CA: Sage.
Teachers of English to Speakers of Other Languages,
Inc. (2001). Qualitative research guidelines.
TESOL Quarterly, 35, 219220.
Wieder, D. L. (1998). Ethnomethodology, conversation
analysis, microanalysis, and the ethnography of
speaking (EMCAMAES): Resonances and ba-
sic issues. Research on Language and Social Interac-
tion, 32, 163171.
Wong, J. (2000). Delayed next turn repair initiation in
native/nonnative speaker English conversation.
Applied Linguistics, 21, 244267.
APPENDIX A
Qualitative Research Guidelines from TESOL Quarterly
To ensure that Quarterly articles model rigorous qualitative research, the following guidelines are provided.
Conducting the study. Studies submitted to the Quarterly should exhibit an in-depth understanding of the philo-
sophical perspectives and research methodologies inherent in conducting qualitative research. Utilizing these
perspectives and methods in the course of conducting research helps ensure that studies are credible, valid, and
dependable rather than impressionistic and superficial. Reports of qualitative research should meet the following
criteria.
1. Data collection (as well as analyses and reporting) aims at uncovering an emic perspective. In other words, the
study focuses on research participants perspectives and interpretations of behavior, events, and situations rather than
etic (outsider-imposed) categories, models, and viewpoints.
2. Data collection strategies include prolonged engagement, persistent observation, and triangulation. Researchers
should conduct ongoing observations over a sufficient period of time so as to build trust with respondents, learn the
culture (e.g., of the classroom, school, or community), and check for misinformation introduced by both the
researcher and the researched. Triangulation involves the rise of multiple methods and sources such as participant
observation, informal and formal interviewing, and collection of relevant or available documents.
Analyzing the data. Data analysis is also guided by the philosophy and methods underlying qualitative research
studies. Researchers should engage in comprehensive data treatment, analyzing data from all relevant sources. In
addition, many qualitative studies demand an analytic inductive approach involving a cyclical process of data
collection, analysis (taking an emic perspective and utilizing the descriptive language the respondents themselves
use), creation of hypotheses, and testing of hypotheses in further data collection.
Reporting the data. Researchers should generally provide thick description with sufficient detail to allow readers
to determine whether transfer to other situations can be considered. Reports also should include
1. a description of the theoretical or conceptual framework that guides research questions and interpretations;
2. a clear statement of the research questions;
3. a description of the research site, participants, procedures for ensuring participant anonymity, and data collec-
tion strategies and a description of the roles of the researcher(s);
4. a description of a clear and salient organization of patterns found through data analysisreports of patterns
should include representative examples, not anecdotal information;
5. interpretations that exhibit a holistic perspective, in which the author traces the meaning of patterns across all
the theoretically salient or descriptively relevant micro- and macrocontexts in which they are embedded;
6. interpretations and conclusions that provide evidence of grounded theory and discussion of how the theory
relates to current research/theory in the field, including relevant citationsin other words, the article should
focus on the issues or behaviors that are salient to participants and that not only reveal an in-depth under-
standing of the situation studied but also suggest how it connects to current related theories.
Note. From TESOL Quarterly Spring 2001, pp. 219220. Copyright 2002 TESOL. Reprinted with permission.
Anne Lazaraton 11
APPENDIX B
Review Checklist for Ethnographic Method (Stewart, 1998)
A. Veracity
1. Prolonged fieldwork
2. Seeking out reorienting or disconfirming observations
3. Good participative role relationships
4. Attentiveness to speech and interactional contexts
5. Multiple modes of data collection
B. Objectivity
1. Trail of the ethnographers path
2. Respondent validation
3. Feedback from outsiders
4. Interrater checks on indexing and coding
5. Comprehensive data archive
C. Perspicacity
1. Intense consideration of the data
2. Exploration
Note. From The Ethnographers Method (pp. 6874), by A. Stewart, 1998, Thousand Oaks, CA: Sage. Copyright 1998 by
Sage, Inc. Reprinted by permission of Sage Publications, Inc.
APPENDIX C
Criteria for (Ethical/Literary/Scientific) Ethnography (Richardson, 2000)
1. Substantive contribution
Does this piece contribute to our understanding of social-life?
Does the writer demonstrate a deeply grounded (if embedded) human-world understanding and perspective?
How has this perspective informed the construction of the text?
2. Aesthetic merit
Does the piece succeed aesthetically?
Does the use of creative analytical processes open up the text, inviting interpretive responses?
Is the text artistically shaped, satisfying, complex, and not boring?
3. Reflexivity
How did the author come to write this text?
How was the information gathered?
Ethical issues?
How has the authors subjectivity been both a producer and a product of this text?
Is there adequate self-awareness and self-exposure for the reader to make judgments about the point of view?
Do authors hold themselves accountable to the standards of knowing and telling the people they have studied?
4. Impact
Does this affect me? emotionally? intellectually? generate new questions? move me to write? move me to try new
research practices? move me to action?
5. Expresses a reality
Does this text embody a fleshed-out, embodied sense of lived-experience?
Does it seem truea credible account of a cultural, social, individual, or communal sense of the real?
Note. From Evaluating Ethnography, by L. Richardson, 2000, Qualitative Inquiry, 6, p. 254. Copyright 2000 by Sage,
Inc. Reprinted with permission.
12 The Modern Language Journal 87 (2003)