Você está na página 1de 9

Systematic reviews of

qualitative research
D. Evans and A. Pearson

Systematic reviews have become an important component of evidence-based health

care. To date, these reviews have focused predominantly on randomized controlled
trials to determine the effectiveness of treatments. However, there is a growing interest
in reviews of qualitative research to summarize research addressing a phenomenon of
interest or to supplement evidence generated by randomized controlled trials. These
reviews have the potential to make an important contribution because they can bring
the person s perspective in to the decision making process. Like all systematic reviews,
optimal methods for reviewing qualitative research are still evolving, however published
reviews demonstrate considerable variability in their approach. Despite their potential
usefulness there has been little professional discussion and debate. This paper presents
the current methods used for the review of qualitative studies and highlights some areas
in need of further exploration. °C 2001 Harcourt Publishers Ltd

INTRODUCTION than the RCT. For example, because qualitative

studies explore the beliefs, expectations and un-
During the past two decades there has been a grow- derstandings of patients, they have the potential to
ing interest in the scientific basis of health care influence health care decisions.
interventions. In conjunction with this, the sys- Despite this potential, qualitative evidence has
tematic review has risen to prominence as means not played a significant role in the evidence-based
to identify, appraise and then synthesize research. movement. Part of the reason for this is the exclu-
These reviews have focused predominantly on ran- sion of qualitative studies from systematic reviews.
domized controlled trial (RCT) to determine the With the increasing use of systematic reviews as
effectiveness of interventions. This pivotal role of the basis for health care decisions, this exclusion
the systematic review is reflected in the consid- has meant that qualitative research has been ex-
erable professional discussion, development and cluded from the decision making process.
refinement of review methods, addressing issues Despite this marginalization of qualitative re-
such as search strategies (Adams et al. 1992; search, there is a growing interest in the synthesis
Counsell 1997; Dickersin 1994), critical appraisal of data generated by individual qualitative studies.
(Jadad et al. 1996; Moher et al. 1995; Sindhu et al. It has been suggested that by combining quali-
David Evans RN. 1997) and meta-analysis (DeMets 1987; Dickersin tative studies in a review, findings become more
DipN. BN. MNS, and Berlin 1992; LeLorier et al. 1997). significant because they draw on broader range
Coordinator of Reviews of participants and descriptions (Sherwood 1999).
The Joanna Briggs However, despite all this activity, the scientific
Institute for Evidence basis of health care has been narrowly defined. These reviews consolidate the body of knowledge
Based Nursing and on a topic, organizing and describing the charac-
Midwifery This is a consequence of measuring the value of
interventions primarily by effectiveness. As a re- teristics of studies, their methods and the findings.
Alan Pearson RN PhD
Professor of Nursing sult, the RCT is commonly seen as the gold stan- As a result of this process a new interpretation of
La Trobe University dard because it is the optimal method to explore the phenomenon is produced, contributing to the
Correspondence to: issues related to cause and effect. This is not to understanding of the topic. For example, these re-
David Evans, The Joanna views allow exploration of research addressing the
Briggs Institute, Margaret suggest that knowing whether interventions work
Graham Building, The is not vital, rather that it is also important to know impact of illness, patients experiences with health
Royal Adelaide Hospital, care, or organizational aspects of care delivery.
North Terrace, whether interventions should and can be used.
Adelaide, South Some of the evidence needed to answer ques- These reviews can also contribute contextual in-
Australia, Australia 5000; formation to supplement evidence generated by
Tel.: (61) 8 8303 4880; tions about the appropriateness and feasibility of
E-mail: interventions can be generated by methods other RCTs.

Clinical Effectiveness in Nursing (2001) 5, 111–119 °

C 2001 Harcourt Publishers Ltd

doi:10.1054/cein.2001.0219, available online http://www.idealibrary.com on

112 Clinical Effectiveness in Nursing

This interest has been matched by a growing question must be broad enough to be of interest but
discussion in the health care literature addressing small enough to be manageable (Sherwood 1999).
the synthesis of qualitative research (Estabrooks Delineating a clear focus is particularly important
et al. 1994; Jensen and Allen 1996; Popay et al. with qualitative reviews because the data is nar-
1998; Sandelowski et al. 1997; Sherwood 1999). A rative text, which does not lend itself to concise
small number of reviews of qualitative studies have summary and synthesis in the same manner as nu-
been published addressing issues such as chronic merical data. Too broad a review question will sig-
illness (Thorne and Paterson 1998), the concept of nificantly increase the time and work required to
caring (Sherwood 1997), the concept of wellness complete the review. However there is little guid-
and illness (Jensen 1994) and stressors in fami- ance on how best to strike this balance. It is likely
lies with a child with a chronic condition (Ogden- that the principles used to guide the primary re-
Burke et al. 1998). However, methods for review- searcher will be equally useful by the reviewer.
ing, summarizing and synthesizing qualitative There are fundamental differences in the focus
research are still evolving and optimal approaches of reviews of RCT compared to those involving
have yet to be determined. This paper presents qualitative research. Meta-analysis aims at provid-
the current approaches used for qualitative reviews ing definitive evidence on the effectiveness of an
and a discussion of related issues. intervention rather than increasing our understand-
ing of a phenomenon. Yet each approach provides
vital information for practice. To date these two
REVIEWS OF QUALITATIVE approaches have been seen as mutually exclusive.
RESEARCH That is, reviews focus on either RCT or qualitative
research, never both. Yet both provide a unique fo-
Noblit and Hare developed an approach for the cus, and so perhaps an opportunity exists to com-
synthesise the findings of ethnographic and other bine approaches to provide a richer source of ev-
types of qualitative studies that they termed meta- idence on which to base practice. The evidence
ethnography (Noblit and Hare 1988). More re- generated from this type of review would inform
cently the broader term meta-synthesis has come practice not only on the effectiveness of an inter-
into use (Jensen and Allen 1996; Sandelowski vention, but also what impact it has on the person.
et al. 1997; Sherwood 1999), referring to the syn- However, this potential has yet to be explored.
thesis of findings from different types of qualitative
research. Sherwood describes meta-synthesis as Locating studies
the critical review and analysis of processed data,
or the themes, domains and descriptions generated Systematic reviews of RCT attempt to identify all
by multiple qualitative studies (Sherwood 1999). work on the topic of interest and this is reflected in
The goal of meta-synthesis differs significantly attempts to improve search strategies (Adams et al.
from meta-analysis, being interpretive rather than 1992; Counsell 1997; Dickersin 1994; Hay et al.
aggregative, it seeks to generate understanding or 1996; Jadad and McQuay 1993). This approach
interpretive explanations of a phenomenon (Jensen is consistent with RCT methods that aim to con-
and Allen 1996). Through the process of synthesis, trol variables, use objective outcome measures and
the review allows critical examination of multiple ensure complete follow up and accounting of all
accounts of an event or situation, a systematic com- participants. However, it is not clear whether re-
parison of studies, a way of taking about specific views of qualitative research should adopt a sim-
work and comparing it to the work of others, and ilar approach or use a sample of the total number
synthesis of qualitative research (Noblit and Hare of available studies. Additionally, if only a sam-
1988). ple of the literature is to be used, the size of the
However, the optimal methods for reviewing sample to ensure an adequate understanding of the
this research are still unclear and approaches em- phenomenon is difficult to determine (Jensen and
ployed in published reviews differ. While the steps Allen 1996).
used during the review of RCT, such as formulating An exhaustive search of the literature, as is done
a question, locating the studies, critical appraisal, during systematic reviews, would ensure that all
data extraction and data synthesis, also apply to re- relevant studies are included in the review. This
views of qualitative research, there are significant approach would help prevent important informa-
differences in their application. tion being excluded from the review. Additionally,
the findings would be strengthened because of the
broader base from which the descriptions were
generated. However, for topics with a large number
of published studies, attempting to summarise and
synthesize 100 or more qualitative studies would
Defining the focus
be a vast and time consuming undertaking.
Like systematic reviews, reviews of qualitative re- The alternative approach of using only a sam-
search seek to answer a specific question. The ple of available studies would make it possible
Systematic reviews of qualitative research 113

to generate a rich and detailed text on the phe- • Reporting of themes and labels must be clear
nomenon. This type of review would not aim to and must be grounded in the data to be able to
identifying every issue on the topic, rather it would demonstrate comparability of themes and
generate a text documenting the major themes and labels across studies.
categories from the sample of studies. In support
of this approach, Noblit and Hare (Noblit and Hare Evaluating published reviews of qualitative re-
1988) argue that unless there is some substan- search, not all state the criteria used for study selec-
tive reason for an exhaustive search, generalisa- tion and few provided full details. Of the reviews
tion from all studies on a particular topic yields that cited information about study selection, issues
trite conclusions. However, with the use of a repre- addressed by the criteria included:
sentative sample of studies, information that could • Clear description of study participants or that
make an important contribution to the understand- the study used a specific group of participants
ing of the topic may be omitted from the review. (Bunting 1997; Ogden-Burke et al. 1998;
Examining published reviews reveals a lack of Thorne and Paterson 1998)
consistency in approaches used to review qualita- • The study focused on a specific issue such as
tive studies. Some reviewers attempted to find all illness or investigated the experience of living
past research on a topic (Sherwood 1997; Thorne with a condition such as diabetes or chronic
and Paterson 1998), others placed publication date illness (Jensen 1994; Paterson et al. 1998;
limits on the search (Jensen 1994; Ogden-Burke Thorne and Paterson 1998)
et al. 1998; Paterson et al. 1998), while one re- • The report provided enough description of the
viewer chose a sample of studies, recent articles method to infer qualitative approach was used
and older classics (Bunting 1997). (Ogden-Burke et al. 1998; Paterson et al.
While it appears that reviews of qualitative 1998)
research can be based on either a comprehensive • Primary researchers were nurses (Bunting
or representative sample of studies, the impact of 1997)
this decision on review findings is not clear. For • The study was published within specific time
topics with a limited number of studies, a com- frame (Thorne and Paterson 1998)
prehensive approach would be the obvious choice. • The study was reported in English (Thorne
For topics with a substantive body of research, a and Paterson 1998), and
representative sample may be one way in which • Sufficient evidence of data trail was provided
to deal with large numbers of studies. While there (Thorne and Paterson 1998).
have been no comparisons of the two approaches,
the summary provided by each would likely differ. Based on information provided in these re-
Reviews that use only a sample of available studies views, the most common criteria used for study
would be at considerably greater risk of missing selection addresses participants, study focus and
information that could make an important contri- description of qualitative method. Sufficient ev-
bution to the understanding of the phenomenon. idence of a data trail has also been highlighted
as an important issue by Thorne (Thorne and
Paterson 1998) and Estabrooks (Estabrooks et al.
Selecting studies for inclusion 1994). However, unlike criteria used to select RCT
in review for inclusion in systematic reviews, there appears
to be little agreement on the optimal components
Criteria of eligibility are used during the conduct of
of criteria for the selection of qualitative studies.
systematic reviews to ensure only studies address-
Additionally, the reporting of the criteria used is at
ing the specific topic of interest are included in
times inadequate, and this issue has received only
the review. These criteria are based on clear scien-
limited attention, suggesting further investigation
tific reasoning and are developed and documented
and clarification is needed.
prior to commencement of the review (Mulrow and
Oxman; NHS Centre for Reviews and Dissemi-
nation 1996; Petitti 1994). When the focus of the Critical appraisal
systematic review is RCT, the inclusion criteria de-
scribes the population, intervention and outcome Prior to inclusion in a systematic review, RCT are
measures of interest. When selecting qualitative critically appraised to exclude any studies whose
studies for inclusion in a review Estabrooks et al. results are at risk of error or bias. Whether this
(Estabrooks et al. 1994) suggest that criteria of process should be applied to reviews of qualitative
importance include: research is less clear. Critical appraisal of quali-
tative research has been the subject of consider-
• Studies should focus on similar populations or able controversy and discussion (Engel and Kuzel
themes 1992; Koch 1994; Lincoln 1995; Sandelowski
• Studies should have a similar research 1993; Schwandt 1996; Smith 1987), and its role
approach, as mixing methods may lead to in reviews has yet to be adequately addressed.
difficulty in developing theory Estabrooks et al. (Estabrooks et al. 1994) warn of
114 Clinical Effectiveness in Nursing

the risk of including studies with superficial anal- that did not give details of the analytic procedures
ysis or research biases and they suggest that these were also included in the review (Ogden-Burke
weak studies should stand out as an anomaly or et al. 1998).
because supporting data is missing. While this While the debate continues on how best to ap-
critical appraisal of qualitative studies has been praise qualitative research, minimum criteria do
supported and used in reviews (Sherwood 1999; emerge from completed reviews and the related
Sherwood 1997), others argue against appraisal literature. A clear description of method and par-
(Jensen 1994; Jensen and Allen 1996). ticipants is vital. Without this description, any at-
Even the term qualitative adds to this difficulty tempt at summary and synthesis of findings would
because it is used to represent a range of different be difficult. An inadequate description would also
methods that are commonly categorized together. increase the risk of synthesizing dissimilar stud-
These differing methods are often not differen- ies. Equally important is evidence of a data trail.
tiated when issues related to scientific rigour are Estabrooks et al. note that when examining quali-
addressed. Additionally, Burns argues that the ma- tative research findings, it is critical that labels are
jor obstacle to the identification of excellence in closely and explicitly tied to supportive data or ex-
qualitative studies is attempts to evaluate these emplars (Estabrooks et al. 1994). This is because
studies using quantitative criteria (Burns 1989). categories or themes used across different studies
Criteria developed by Burns to appraise qualitative may not always have the same label.
studies, addresses all types of qualitative research However, the issue of how best to critically
(Burns 1989): appraise research is not limited to qualitative re-
search, as witnessed by the large number of check-
1. Descriptive Vividness lists and scales that have been developed for the
2. Methodological Congruence appraisal of RCT (Moher et al. 1995). This sug-
• Rigour in documentation gests that critical appraisal of all research is an
• Procedural rigour emerging field and will continue to grow and de-
• Ethical rigour velop in the future.
• Auditability.
3. Analytical Preciseness Data extraction
4. Theoretical Connectedness The aim of this phase of systematic reviews is to
5. Heuristic Relevance collect data from studies that are reliable, valid
• Intuitive recognition and free from bias (Petitti 1994). To this end, a
• Relationship to existing body of knowledge data collection form is used to bridge the gap be-
• Applicability. tween what is reported by the primary investiga-
tor and what is ultimately reported by the reviewer
The value of this tool as part of the system- (Mulrow and Oxman 1997). Information collected
atic review process is not clear, but it is likely that during reviews of RCT include such things as bibli-
different appraisers using this tool would reach ographic details, description of study setting, pop-
differing conclusions. ulation, details on delivery of intervention, out-
From the perspective of the reviewer, standard- come measures, study method, any factor that may
ized criteria would aid in the conduct of reviews influence the validity of findings and the actual re-
of qualitative research. However, to date there has sults of the research (NHS Centre for Reviews and
been no agreement on these criteria. In evaluating Dissemination 1996).
the approach to critical appraisal, as reported in During reviews of qualitative studies the data
published qualitative reviews, there is great vari- collection phase is less clear and is usually reported
ation in the methods used and little consistency. as being part of the synthesis process, during the
During a review of research addressing chronic reading and rereading of the text (Jensen 1994;
illness, Thorne and Paterson appraised all studies Noblit and Hare 1988; Paterson et al. 1998). The
to assess the rigour, epistemologic soundness, and text of individual studies is treated more as a whole
fruitfulness of the research methods used (Thorne during initial readings while major themes and cat-
and Paterson 1998). A review addressing wellness- egories are identified, and then compared and con-
illness placed no restriction on the scientific merit trasted with those of other studies. This represents
of studies to avoid eliminating data germane to the a significant departure from the approached used
purpose of the investigation (Jensen 1994). One re- with RCT, where collection of data is a distinct
view, focusing on how people adapted to the man- phase of the review, involving checks to maintain
agement of their diabetes, failed to mention critical rigour and pilot testing of forms.
appraisal or study quality in the description of the However, the use of coding sheets have been
review method (Paterson et al. 1998). Another re- recommended to help track study details such
view addressing stressors in families with a child as research design, assumptions and sample
with chronic illness, also did not appraise studies, (Sherwood 1999). Jensen and Allen recommended
and noted that studies published in clinical journals listing key metaphors, phrases, concepts, ideas and
Systematic reviews of qualitative research 115

categories, although this is part of the synthesis While heterogeneity of results is a cause for
process rather than a distinct data collection phase concern during meta-analysis, greater variation
(Jensen and Allen 1996). Additionally, some re- in findings should be anticipated from qualita-
views provide details of study characteristics in the tive studies. However, these differences are not
form of tables (Jensen 1994; Ogden-Burke et al. merged, rather these differences are compared
1998; Sherwood 1997), suggesting that specific and contrasted, and areas of commonality iden-
data was collected at some stage of the review tified as part of the synthesis (Paterson et al. 1998;
process. Sherwood 1997). The aim of this synthesis is
One review of qualitative research address- to portray an accurate interpretation of the phe-
ing chronic illness identified over 400 studies nomenon, and to compare and contrast the con-
of which 158 met the review inclusion criteria structs of individual studies to reach consensus on
(Thorne and Paterson 1998). Clearly, some top- a new construction of the phenomenon (Sherwood
ics have a large body of research, and that deal- 1999). Studies are translated into one another,
ing with this volume of information would be which involves comparing both the metaphors or
impossible without a formalized process of col- concepts and the interaction of one account with
lection and recording of relevant information and those of other accounts (Jensen and Allen 1996;
study characteristics. Additionally, reporting the Sherwood 1999). This results in a progressive re-
characteristics of studies would be an important finement of the understanding of the phenomenon
component of the final review report, and would through a process of constant comparative analy-
represent part of the reviews data trail that has sis. The aim is to generate a new construction on
been claimed to be so important for primary which there is consensus (Jensen and Allen 1996).
research. However, this component of reviews Differing research methods, such as phe-
of qualitative studies has received only limited nomenology, ethnography or grounded theory, are
attention. not mixed in a single synthesis of all qualitative
studies. If different qualitative research methods
are mixed, it will be unclear what has been
Data synthesis
obtained from the synthesis. (Jensen and Allen
The analysis and synthesis of qualitative stud- 1996). The basis for this is that while qualitative
ies is commonly termed meta-synthesis, and like studies using differing methods may address the
meta-analysis, it is based on processed data. How- same topic, each method provides its own unique
ever, there are major differences between the ap- perspective.
proach used to synthesise the findings of RCT As qualitative studies differ from RCT, meta-
and qualitative studies. Reality for the qualita- synthesis also uses an approach that is markedly
tive researcher, and reviewer, is viewed as mul- different from that used during meta-analysis.
tiple and constructed (Sandelowski 1993), and However, both approaches provide an understand-
so undertaking meta-synthesis means that no two ing that is based on a range of populations, settings
reviewers will produce exactly the same results and circumstances. This broad base for generation
(Jensen and Allen 1996). While meta-synthesis of evidence on a phenomenon allows for greater
provides only one interpretation, it aims to cap- confidence in the evidence. However, unlike meta-
ture the essence of the phenomenon of interest analysis, meta-synthesis deals in multiple realities
(Sherwood 1999). and so it provides but one interpretation of the
First stage of the synthesis involves reading phenomenon.
and re-reading of texts from the individual stud-
ies, noting interpretive metaphors, themes and
categories. This phase continues throughout the DISCUSSION
duration of the synthesis (Jensen and Allen 1996;
Sherwood 1999). Data is standardized through While reviews of qualitative research have the po-
common codes and outlines, and it has been sug- tential to make an important contribution to nurs-
gested that a table summarizing the bibliographic ing knowledge some issues have yet to be resolved,
information and other characteristics of studies and like systematic reviews, it is a method that is
helps display details for comparison and com- still evolving. The published reviews are impor-
pilation (Sherwood 1999). Studies are sorted by tant in that they represent exploration of not only
methodology, data collection instrument and data the review topic, but also of potential review meth-
analysis methods (Sherwood 1999), and by key ods. These reviews provide a basis for continued
metaphors, phrases, ideas and concepts (Jensen development of methods and help focus profes-
and Allen 1996). This process allows data to be sional debate. However, to date this professional
reduced to common denominators, as is done dur- debate has been quite limited.
ing meta-analysis (Sherwood 1999). Studies are Of the reviews published, some provide only
examine for homogeneity and the initial assump- limited information of the processes and meth-
tions about the relationships between studies are ods used. For example, information related to how
developed (Sherwood 1999). studies were identified, assessed for suitability to
116 Clinical Effectiveness in Nursing

the review and critically appraised, is often inad- However optimal methods have yet to be deter-
equate. This limits the opportunity to appraise the mined and published reviews demonstrate some
trustworthiness of the review itself. This is a sig- variability in the approach to the summary and
nificant departure from reporting of systematic re- synthesis of studies. However, while the methods
view methods, which aim to provide sufficient in- used to review qualitative research are still evolv-
formation to permit the assessment and appraisal ing, professional discussion and debate have been
of the methods used. As an emerging field, this limited. Issues such as what questions can best be
complete description is also important for future addressed by this approach, and the role of these
endeavours, in that it allows continued refinement reviews within the broader evidence-based health
and development of review methods. This lack of care movement, have yet to be fully explored. Fi-
information, combined with the occasional poor nally, qualitative reviews are an emerging field,
description of the findings of the review, means and therefore the published reviews are important
that the data trail that is vital for qualitative study in that they provide the basis for on going devel-
reports is lacking from review reports. opment and refinement of methods.
It is not clear what questions can reasonably
be addressed by this type of review. While it is
likely that suitable questions for the primary re- REFERENCES
searcher would also be appropriate for the re-
viewer, this issues has not been fully explored. Adams CE, Lefebvre C, Chalmers I 1992 Difficulty with
The validity of the findings of qualitative reviews MEDLINE: searches for randomised controlled trials.
also requires further exploration. These findings The Lancet 340: 915–916
Bunting SM 1997 Applying a feminist analysis model to
are based on the third level of interpretation of the selected nursing studies of women with HIV. Issues in
phenomenon, that is, the event or phenomenon is Mental Health Nursing 18: 523–537
interpreted by the person, the researcher and finally Burns N 1989 Standards for qualitative research. Nursing
by the reviewer. A concern is that reviews may in- Science Quarterly 2: 44–52
crease bias by using these interpretations rather Counsell C 1997 Formulating questions and locating primary
studies for inclusion in systematic reviews. Annals of
than the original informants descriptions (Jensen Internal Medicine 127: 380–387
and Allen 1996). The risk is that these reviews pro- DeMets DL 1987 Methods for combining randomised clinical
duce readable, but meaningless, descriptions that trials: strength and limitations. Statistics in Medicine 6:
are too far removed from the original event to be of 341–348
any value for generating an understanding about a Dickersin K, Scherer R, Lefebvre C 1994 Identifying relevant
studies for systematic reviews. British Medical Journal
phenomenon. 309: 1286–1291
The full potential of this type of review has Dickersin K, Berlin JA 1992 Meta-analysis: state of the
yet to be realised. While they are currently used science. Epidemiologic Review 14: 154–176
to increase the knowledge and understanding on Engel JD, Kuzel AJ 1992 On the idea of what constitutes
a phenomenon, such as the experience of chronic good qualitative inquiry. Qualitative Health Research
2(4): 504–510
illness or management diabetes, they could also LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian
provide vital information that is complementary F 1997 Discrepancies between meta-analyses and
to that provided by systematic reviews of RCT. subsequent large randomised controlled trials. New
From this perspective, reviews of qualitative re- England Journal of Medicine 337: 536–542
search could complement the answer to the ques- Lincoln YS 1995 Emerging criteria for quality in qualitative
and interpretive research. Qualitative Inquiry 1(3):
tion does this intervention work provided by the 275–289
RCT, with answers to should it be used or what Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh
does this mean to the person. S 1995 Assessing the quality of randomized controlled
These reviews have potential to bring the pa- trials: an annotated bibliography of scales and checklists.
tient’s perspective into the decision making pro- Controlled Clinical Trials 16: 62–73
Mulrow CD, Oxman AD Cochrane Collaboration Handbook
cess. For example, issues such as chronic pain, [updated 1 March 1997], In: The Cochrane Library
incontinence or the fear of falling can be acknowl- [database on disk and CDROM]: The Cochrane
edged as being important because of their impact Collaboration, Oxford: Update Software
on a person’s life. Qualitative research should play NHS Centre for Reviews and Dissemination 1996
an important role in the evidenced based health Undertaking Systematic Reviews of Research on
Effectiveness: CRD Guidelines for Those Carrying Out
care movement, influencing both clinical practice or Commissioning Reviews. York: University of York
and policy decisions. However, in the current ap- Noblit GW, Hare RD 1988 Meta-ethography: Synthesizing
proach to health care, this research has had only a Qualitative Studies. Vol. 11. Newbury Park: Sage
peripheral role. Publications
Ogden-Burke S, Kauffmann E, Costello E, Wiskin N,
Harrison MB 1998 Stressors in families with a child with
a chronic condition: analysis of qualitative studies and a
framework. Canadian Journal of Nursing Research
30(1): 71–95
Paterson BL, Thorne S, Dewis M 1998. Adapting to and
Reviews of qualitative research have the potential managing diabetes. Image 30(1): 57–62
to make an important contribution to health care. Petitti DB 1994 Meta-analysis, decision analysis and
Systematic reviews of qualitative research 117

cost-effectiveness analysis. New York: Oxford Sherwood G 1999 Meta-synthesis: merging qualitative
University Press studies to develop nursing knowledge. International
Popay J, Rogers A, Williams G 1998 Rationale and standards Journal of Human Caring 3(1): 37–42
for the systematic review of qualitative literature in Sherwood GD 1997 Meta-synthesis of qualitative analyses of
health services research. Journal of Qualitative Health caring: defining a therapeutic model of nursing.
Research 8(3) 341–51 Advanced Practice Nursing Quarterly 3(1): 32–42
Sandelowski M 1993 Rigor or rigor mortis: the problem of Sindhu F, Carpenter L, Seers K 1997 Development of a tool
rigor in qualitative research revisited. Advances in to rate the quality assessment of randomized controlled
Nursing Science 16(2): 1–8 trials using a Delphi technique. Journal of Advanced
Sandelowski M, Docherty S, Emden C 1997 Qualitative Nursing 25: 1262–1268
metasynthesis: issues and techniques. Research in Smith ML 1987 Publishing qualitative research. American
Nursing and Health 20: 365–371 Educational Research Journal 24: 173–182
Schwandt TA 1996 Farewell to criteriology. Qualitative Thorne S, Paterson B 1998 Shifting images of chronic illness.
Inquiry 2(1): 58–72 Image: Journal of Nursing Scholarship 30(2): 173–178

D. Evans and A. Pearson Clinical Effectiveness in Nursing (2001) 5, 111–119
Systematic reviews of qualitative research

This paper outlines the major theoretical and [albeit limited] empirical positions in the debate sur-
rounding the role of qualitative research in systematically produced reviews of evidence. Good quality
qualitative research with its rich, deep, meaningful description of a phenomenon or events is ideally
placed to fill the information gaps in decisions involving the experiences of patients receiving healthcare.
The authors recognise the value of such understanding to decision making but then go on to highlight
some of the problems associated with systematically reviewing qualitative research. Their picture is
somewhat bleak and I would like to propose a more optimistic response – borne largely of pragmatism.
Firstly, I do not agree that qualitative research has been excluded from the evidence based healthcare
agenda. The United Kingdom Health Technology Assessment (HTA) programme has published reviews
of the role of qualitative research in the evaluation of health technologies (Murphy et al. 1998) and has
recently commissioned a study into ways of developing the role of qualitative research in systematic
reviews. Moreover, both the International Cochrane (for healthcare) and Campbell (for social policy)
collaborations have methods groups concerned with the role of qualitative research in systematic reviews
of effectiveness. The limited role of qualitative research in evidence based healthcare may not be due to
the myopia of commissioners and policy makers but rather the nihuistic responses qualitative researchers
sometimes entertain when having to demonstrate the quality of their product (c.f. Barbour 2001).
Secondly, the authors rightly point out that the science of reviewing qualitative research is in its
infancy and that significant obstacles to synthesizing such knowledge exist. The problem of unfo-
cused reviews could be overcome by using clinical rather than research questions as the basis for
reviews. Focused clinical questions can be both a useful filter for the kinds of research material sought
and increasing end-user relevance. Locating studies of qualitative research is difficult, but tools such
as the qualitative search filter at the University of Rochester (http://www.urmc.rochester.edu/miner/
educ/ebnfilt.htm) are making the task easier. However, research needs to be carried out ascertain the
sensitivity and specificity of such search tools. On the difficulties involved in establishing criteria for
inclusion in reviews of qualitative research I would concur. The task here is for primary researchers to
develop and report the choices they make at the sampling stage. It is when one attempts to critically
appraise qualitative research, however, that the conflict between the needs of decision makers and the
qualitative research community reaches its zenith. Scholars may have the luxury of being able to debate
the merits and demerits of various quality criteria, but clinician systematic reviewers are not so lucky.
However, there are various quality appraisal tools in existence which can at least make the criteria used
for appraisal transparent. The Critical Appraisal Skills Programme (CASP) qualitative checklist is a
good example (http://www.publichealth.org.uk/casp/qualitative.html). As long as readers of reviews can
identify the choices made then where is the problem? Whilst (imperfect) reviews are being undertaken,
the scholars can then busy themselves with refining the science of appraisal, and reviewing qualitative
evidence will develop incrementally. Other pragmatic responses to the authors concerns could be offered
but the take home message, for this reader at least, should be that efforts at reviewing and synthesizing
qualitative evidence should continue apace. In this way raw materials for the research community to
scrutinize and improve upon will be generated.
This is an important and cogent paper, offering a brief tour of some of the major dilemmas in this
area of research synthesis. The end result of this overview is a solid foundation for the development of
further (empirically informed) debate on this topic.
118 Clinical Effectiveness in Nursing


Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 322:
Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology
assessment: a systematic review of the literature. Health Technology Assessment 1998; 2(16).

Carl Thompson
doi:10.1054/cein.2001.0224, available online http://www.idealibrary.com on

D. Evans and A. Pearson Clinical Effectiveness in Nursing (2001) 5, 111–119

Systematic reviews of qualitative research

This paper is a timely attempt to examine the place of systematic reviews in qualitative research.
Certainly the possibilities of such reviews are not currently widely recognized. However, there are a
number of issues I would raise for the authors to think about.
The first point, only partly addressed by Evans and Pearson in their conclusion, is that the whole
point of qualitative research is to dig deep, to examine rich detail and to gain insight into unique human
experiences. To collect, aggregate and summarize studies which report these things must necessarily
take away from these aims and I can even imagine reviewers counting instances of this of that category,
concept or phenomenon in complete contradistinction to the true aims of most, if not all qualitative
enterprise. Whilst such review activities might have some basic value as background, the methodological
fetishism which some researchers have adopted would certainly be counterproductive in this case.
My second point is that systematic reviews are generally much less critical than they might at first
seem. The mass aggregation of outcomes and bare facts from large numbers of studies defeats the
possibility of appropriate criticism except in the most abstract of terms. The way to analyze and criticise
research papers is to allocate the process a lot of words for each criterion against which the study is
to be evaluated. To come to these criteria, Evans and Pearson give a good summary of those in some
use today and they recognize that many are unsatisfactory. It seems to me that few of those criteria
mentioned address the real issues. Why ‘primary researcher were nurses’ (Bunting 1997) is important is
hard to see; the important criteria (applicable to all forms of qualitative research) and made reasonably
clear by Martin Hammersley (1990) are validity, relevance and importance. Hammersley shows that
evaluation of qualitative research requires analytic and critical depth of a type not frequently seen in
the nursing research literature and unlikely to prevail in the ‘systematic review’ as described here.
One particular aspect of assessing validity is the approach to sampling. Here I mean not that of the
original papers reviewed, but the selection of papers for inclusion. Evans and Pearson seem to feel that
such a thing as a ‘representative sample’ would be possible, but make no suggestions as to how this will
be achieved. Typically, though thankfully not here, randomness is brought into play as a mechanism for
the achievement, or at least the assumption, of representativeness of the field. It will be important to lay
this particular myth to rest early on in the evolution of qualitative systematic reviews, since random or
other chance methods for inclusion would in all probability have left out the most important works of all.
My final concern in this vein is in the suggestion, not contradicted by Evans and Pearson, that it
will be in some way wrong to mix studies which used different methods and perspectives in systematic
reviews. My views on this are put in much greater depth elsewhere (Johnson, Long and White 2001), but
sufficient to argue that my associates and I have found virtually no qualitative research which follows
exactly the prescriptions and procedures of the various exponents of ‘tight’ methods.
Methodological purity is a myth and so there surely can be little wrong in comparing and contrasting
outcomes from the various forms of qualitative research, if it can be established that the review has
value in the first place.
To conclude, the paper makes a valuable contribution to the evaluation of the place of systematic
reviews of qualitative evidence. It may be, however, that the goal of answering specific questions is
misplaced. Rather than providing answers, in my view such reviews will refine old questions and
hopefully ask new ones, surely, a much more important and interesting aim.


Hammersley M 1990 Reading ethnographic research: a critical guide. Longman, London

Johnson M, Long T, White A 2001 Arguments for ‘British Pluralism’ in qualitative health research. Journal of
Advanced Nursing 33(2): 243–249

Martin Johnson
doi:10.1054/cein.2001.0223, available online http://www.idealibrary.com on
Systematic reviews of qualitative research 119

Author’s response
The authors would like to thank the two commentators for their response to this discussion on the
conduct of systematic reviews of qualitative research. The commentators have raised many worthy
points and have presented a balanced and thought-provoking discussion. The authors agree with these
commentaries, and would like to highlight the need for an informed public debate on this important

David Evans
doi:10.1054/cein.2001.0222, available online http://www.idealibrary.com on