Western Journal of Nursing Research

http://wjn.sagepub.com The Challenges of Searching for and Retrieving Qualitative Studies

Julie Barroso, Claudia J. Gollop, Margarete Sandelowski, Janet Meynell, Patricia F. Pearce and Linda J. Collins West J Nurs Res 2003; 25; 153 DOI: 10.1177/0193945902250034 The online version of this article can be found at: http://wjn.sagepub.com/cgi/content/abstract/25/2/153

Published by:
http://www.sagepublications.com

On behalf of:
Midwest Nursing Research Society

Additional services and information for Western Journal of Nursing Research can be found at: Email Alerts: http://wjn.sagepub.com/cgi/alerts Subscriptions: http://wjn.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations http://wjn.sagepub.com/cgi/content/refs/25/2/153

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

Western Journal of Nursing Research, 2003, 25(2), 153-178

Western Journal of Nursing Research March 2003, Vol. 25, No. 2 DOI: 10.1177/0193945902250034

ARTICLE

The Challenges of Searching for and Retrieving Qualitative Studies1

Julie Barroso Claudia J. Gollop Margarete Sandelowski Janet Meynell Patricia F. Pearce Linda J. Collins

The authorspurpose in this article is to report the results of their search and retrieval efforts to date in an ongoing study to develop the procedural, analytic, and interpretive techniques to conduct qualitative meta-synthesis projects, using studies on women with HIV infection as the method case. For researchers conducting qualitative meta-synthesis projects, the ideal goal is to retrieve all of the relevant studies in a fieldnot simply a sample of them. Batess model of berrypicking is used as the framework to describe the techniques used to conduct these searches. The authors discuss, in particular, the challenges of working with bibliographic databases, including choosing which databases to search, learning about the idiosyncrasies of working with each database, developing a list of search terms, and refining inclusion criteria regarding which studies to include in the meta-synthesis. Recommendations are given for searchers and writers of qualitative research. Keywords: qualitative meta-synthesis; qualitative research; research integration; information retrieval; information-seeking behavior; computer literature searching

There has been an unprecedented proliferation of qualitative studies on various aspects of health, illness, and life transitions across the disciplines
Julie Barroso, Ph.D., A.N.P., C.S., Assistant Professor, School of Nursing, University of North Carolina at Chapel Hill; Claudia J. Gollop, Ph.D., Associate Professor, School of Information and Library Science, University of North Carolina at Chapel Hill; Margarete Sandelowski, Ph.D., R.N., F.A.A.N., Cary C. Boshamer Distinguished Professor of Nursing, School of Nursing, University of North Carolina at Chapel Hill; Janet Meynell, M.S.N., R.N., School of Nursing, University of North Carolina at Chapel Hill; Patricia F. Pearce, RN, MSN, CS-FNP, School of Nursing, University of North Carolina at Chapel Hill; Linda J. Collins, M.S.L.S., User Services Librarian, Health Sciences Library, University of North Carolina at Chapel Hill.
DOI: 10.1177/0193945902250034 2003 Sage Publications 153

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

154

Western Journal of Nursing Research

and health-related publication venues. These studies contain important findings about a diverse range of health issues, including, most notably, personal and cultural constructions of disease, prevention, and risk; living with and managing the effects (including the treatment effects) of an array of chronic conditions; and decision making on and responses to beginning- and end-oflife technological interventions. The challenge for qualitative researchers now is no longer finding places to publish their work but rather finding all of the studies that are relevant to producing their work. This challenge is especially compelling for researchers conducting qualitative meta-synthesis projects in which the ideal goal is to retrieve all of the relevant studies in a field not simply a sample of them. The most important threat to the validity of any research integration effort is to fail to conduct a sufficiently exhaustive search (Cooper, 1998). Our purpose in this article is to report the results of our own search and retrieval efforts to date in an ongoing study to develop the procedural, analytic, and interpretive techniques to conduct qualitative meta-synthesis projects, using studies on women with HIV infection as the method case.2 We chose this area of research because a sufficient number of qualitative studies exist to warrant meta-synthesis, and it is a field of great significance to womens health and nursing practice. Qualitative meta-synthesis is a distinctive kind of research integration study in which the findings from completed qualitative studies in a target area are combined. Like phenomenology, ethnography, and grounded theory, the term qualitative meta-synthesis refers to both an interpretive product and the analytic processes by which the findings of studies are aggregated, integrated, summarized, or otherwise put together (Estabrooks, Field, & Morse, 1994; Jensen & Allen, 1996; Kearney, 1998; Noblit & Hare, 1988; Sandelowski, Docherty, & Emden, 1997; Schrieber, Crooks, & Stern, 1997). The aim of qualitative meta-synthesis is to create larger interpretive renderings of all of the studies examined in a target domain that remain faithful to the interpretive rendering in each particular study. Although it can be considered an analog to meta-analysis (Glass, McGraw, & Smith, 1981) in that there is a shared interest in synthesizing empirical studies (Noblit & Hare, 1988, p. 10) and a shared desire to use a systematic, comprehensive, and communicable approach to research integration, qualitative meta-synthesis is not about averaging or reducing findings to a common metric (Wolf, 1986, p. 33). The aim of qualitative meta-synthesis is instead to create larger interpretive renderings of all of the studies examined in a target domain that remain faithful to the interpretive rendering in each particular study.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

155

In the first year of our meta-synthesis project, we focused, in part, on searching for and retrieving the published articles and book chapters, as well as the unpublished theses, dissertations, and conference papers that will ultimately comprise the bibliographic sample for the meta-synthesis study. Near the end of the first year of our project, a healthy young woman died while participating in a clinical trial at a major medical center. The New York Times reported that the chief researcher of this clinical trial had searched the scientific literature and found no evidence that the drug given to her might be unsafe. He missed some reports of safety problems published in the 1950s (Death at the Hands of Science, 2001). This sadly highlights the imperative, in any kind of research, to know what others have published. In a recent report of a meta-ethnography of qualitative research on lay experiences of diabetes and diabetes care, the authors (Campbell et al., in press) noted the difficulties they encountered when trying to retrieve all of the literature relevant to their work. They commented that they had learned of another published meta-ethnography of diabetes only after their own study was completed and that the authors of this other report had identified only three out of the seven papers that they themselves had included in their search. The authors of another recent article on the challenges of retrieving qualitative research noted that MEDLINE does not include the term qualitative as a MeSH term (Dixon-Woods, Fitzpatrick, & Roberts, 2001), thereby hindering retrieval of qualitative research. MeSH (Medical Subject Headings) terms are the controlled vocabulary of words and phrases used to index much of the biomedical literature. Evans (2002) also noted the difficulties in searching electronic databases for qualitative research. We describe here some of the challenges we encountered in searching for and retrieving qualitative studies and offer recommendations for both searchers and writers of qualitative studies that will make this work as fruitful and efficient for their purposes as their resources, expanding information, and changing information technology will permit.

HOW WE BEGAN

A metasynthesis project typically begins by defining the substantive, methodological, and temporal boundaries for study. We began our study in June 2000 with the following inclusion criteria: all reports of qualitative studies on women in the United States with HIV infection published between 1993the year in which the first of these studies appeared as

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

156

Western Journal of Nursing Research

indicated by our search at that time (we have since found an article published in 1991)and March 1, 2003, 2 years before the anticipated end of the metasynthesis project in 2005. At the time of grant writing, we expected to find approximately 35 studies, including theses and dissertations. We defined qualitative research as encompassing a highly diverse array of orientations to and strategies for inquiry that generally concern themselves with how the social world is interpreted, understood, experienced, and produced (Mason, 1996). Data generation and analysis techniques are flexible, reflexive, context-sensitive, and case-oriented. We specifically excluded: (a) qualitative studies in which there are no human participants per se (as, for example, in discourse, qualitative content, semiotic, or other qualitative analyses of media representations of women with HIV infection); (b) qualitative studies about nonseropositive women and their experiences as mothers, partners, relatives, friends, and/or caregivers of seropositive persons; and (c) qualitative adjuncts (such as open-ended questions at the end of a structured questionnaire) to largely quantitative studies. We also excluded journalistic or other nonresearch accounts of HIV-positive women. We created a list of method search terms that we thought would reflect our criteria well and would therefore capture the studies we were seeking. They included, in alphabetical order, case study, constant comparison, content analysis, conversation analysis, descriptive study, discourse analysis, ethnography, exploratory study, feminist, focus group, grounded theory, hermeneutic, interview, narrative/narrative analysis, naturalistic study, participant observation, phenomenology, qualitative method, qualitative research, and thematic analysis. Where appropriate, we truncated the terms to broaden the search.

BROWSING AND BERRYPICKING

As Bates (1989) pointed out, as more resources are brought online, the searcher has a more complex search environment to consider, both in terms of types of sources to use and search techniques to employ with these resources. Browsing can be distinguished from berrypicking in that browsing is less directed and may be useful to stimulate a searchers thinking, but browsing can be an important component of several of the berrypicking strategies we describe below. Bates (1989) argued that berrypicking is closer to the actual behavior of searchers than traditional models of information retrieval. With this model, considered a classic among information

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

157

scientists, at each stage of the search, the searcher is not just modifying search terms to get a better match for a single query. The query itself is continually shifting, in part or whole. This is called an evolving search and is closer to the actual behavior of searchers. The query is satisfied not by a single final retrieved set but by a series of selections of individual references and bits of information at each stage of the ever-changing search. This is not a straight, linear process. Just as a berrypicker would meander through the bushes, looking for clumps of berries (Bates, 1989), the searcher wanders through the information forest, changing directions as needed to follow up on various leads and shifts in thinking. Bates (1989) identified six strategies that are used in berrypicking; we describe these now along with how we actually executed these strategies during our search and retrieval work to obtain qualitative studies of women with HIV infection. We focus in particular on subject searches in bibliographies and abstracting and indexing services, as this was the most complex strategy, and the one most commonly used by searchers.
Footnote Chasing

This technique includes following up on footnotes found in books and articles of interest and therefore moving backward through reference lists; it is also called backward chaining (Bates, 1989). In our project, we reviewed more than 3,600 citations from the reference lists of research reports, and other articles, books, and anthologies on women with HIV infection. We searched the reference lists for anything that was remotely connected to women and HIV infection, even if the actual document being searched was not a research report that might be included in the meta-synthesis.
Citation Searching

Citation searching, or forward chaining, is where the searcher begins with a citation, finds out who cites it by looking it up in a citation database, and thus leaps forward (Bates, 1989). We have not yet done this type of searching as we wanted to have most of what we anticipate will be the final sample prior to completing this step. We also surmised that this may not be a very fruitful technique for us, given how recently most of the articles we are using were published. It usually takes some time for an article to disseminate widely and therefore to be cited by others in more recent works.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

158

Western Journal of Nursing Research

Journal Runs

With this technique, once the researchers have identified a central journal in an area, these journals are systematically hand-searched straight through the relevant volume years (Bates, 1989). Such a technique guarantees complete recall within that journal. We have done this with two journals, Qualitative Health Research and the Journal of the Association of Nurses in AIDS Care. We felt that the first is an excellent source for qualitative work, and the second is a good source for research conducted with HIV-positive people and is the flagship journal of the Association of Nurses in AIDS Care. These two journals, although not precisely central to the area, were the closest to being central. We did not find any articles in either of these journals, using this journal-run technique, that were not also found using bibliographic databases.
Area Scanning

Bates (1989) described this technique as browsing the materials that are physically colocated with materials located earlier in a search, and as a widely used and effective technique. Once we had identified several books that dealt with HIV-positive women, we scanned the shelf locations of these books to find other books that might be of interest to us. We supplemented area scanning with electronic searches as well, searching the electronic card catalog of our library system looking for library materials that could be found by using women and HIV as our subject search terms.
Author Searching

Based on relevant publications found previously, this strategy entails searching by author name to see if an author has done any other work on the same topic (Bates, 1989). We have had good results using this technique, particularly in the iterative searching for dissertations and/or the articles generated from those dissertations. Of the authors whose published work(s) we had already retrieved, we searched by each author name for dissertations, as many of the researchers in the area of women and HIV infection are recent graduates of doctoral programs. We located several dissertations on women with HIV infection using this method. We also searched for articles in every database using each authors name to see if we had not yet retrieved a published article.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

159

Subject Searches in Bibliographies and Abstracting and Indexing Services

This technique, which involves searching bibliographic databases for relevant works in an area, has been the one we have focused on and will be discussed in greater detail. We chose bibliographic databases that were accessible to us, that covered a wide range of disciplines, and that we knew or surmised would yield reports of qualitative studies on women with HIV infection. Our access to these databases was through the library computer network at the University of North Carolina at Chapel Hill. We identified 16 bibliographic databases for our search; these are listed in Table 1, along with the features of each of these databases. The servers we used were primarily OVID (Ovid Technologies), SilverPlatter (Silver Platter Information), and OCLC FirstSearch (OnLine Computer Library Center, Inc.). We have thus far located 71 published and unpublished studies (theses and dissertations) that meet our inclusion criteria, approximately twice the number of studies we expected to find. We have thus far been unsuccessful in obtaining the proceedings from three national conferences on women with HIV infection. Table 2 illustrates the yield of the bibliographic database searches we have completed to date, specifying the number of citations that met the search terms (5,829) and the number of citations that actually met the criteria for inclusion into our bibliographic sample (213), for a total yield of 3.66%. Most citations were found in more than one database, and these numbers represent electronic searches only. The yield of a search is highly related to the task complexity, the user, and the system(s) involved (e.g., Haynes, Wilczynski, McKibbon, Walker, & Sinclair, 1994; Marchionini, 1995; Meadow, Boyce, & Kraft, 2000; Sutcliffe, Ennis, & Watkinson, 2000). The most commonly used performance measures in information retrieval are based on precision and recall. The recall for a set of retrieved documents is the percentage of relevant documents in the database that have been retrieved. The precision, given a set of retrieved documents, is the percentage of documents that have been retrieved that are relevant. Searches may be categorized into two forms based on precision and recall: high-recall searches, in which most or all of the documents on a topic should be retrieved, and high-precision searches, in which the set of documents to be retrieved, often consisting of a small number of documents, is expected to be composed of predominantly relevant documents (Losee, 2000; Marchionini, 1995). In strictly quantitative terms,
(text continues on p. 165)

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

160 TABLE 1: Electronic Databases Selected for Searches

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

Database Name
Academic Search Elite

Content/ Kind of Information

How to Access

Producer or Content Provider

EBSCO Publishing

Thesaurus With Standardized Keywords?

Yes

Standardized Natural Language Language Searching for Searching Qualitative and Boolean Research? Operators?
Yes Yes

Special Features

Dates of Coverage

Scholarly, EBSCOHost multidisciplinary database covering a variety of academic areas Specialized index to AIDS-related research, including clinical and health policy issues

AIDS Information Online (AIDSLINE)

Now incorpo- National Library rated into of Medicine OVID (part of NIH) MEDLINE or PubMed

Yes

No

Anthropological Index Online

Gateway to Web articles for periodicals in the British Museums Mankind Library

Department of Ethnography at the British Museum

No

No

Full text for more than 19841,500 journals; present covers major newspapers and more than 1,800 peer-reviewed journals Yes Journals plus 1980government and present technical reports, meetings abstracts, special publications, AV materials and theses Noincomplete Specialized collection Late Boolean 1950spresent

Anthropological Literature

Black Studies

Cumulative Index to Nursing and Allied Health Literature (CINAHL)

An index to periodicals and edited works emphasis is on archaeology, anthropology (biological, physical, cultural, and social), and linguistics International index to African American, African, and African Diasporan studies Bibliographic references and abstracts on nursing and allied health topics

CD-ROM

Harvard University (Tozzer Library)

Yes

No

Yes

International coverage; includes about 1,000 journals and monographs

1985present

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

CD-ROM

Gale Group

No

No

Yes

Based, in part, on the holdings of the Schomburg Center for Research in Black Culture

1989present

OVID

CINAHL Information Systems

Yes

Yes Yes qualitative studies

Dissertation Abstracts Index (DAI)

Index to dissertations accepted at accredited U.S. Institutions since 1861

OCLC ProQuest FirstSearch Information and Learning

Yes

No

Yes

Journals plus books, conference proceedings, AV materials, educational software, nursing standards, and dissertations Selectively covers masters theses, Canadian dissertations, and British and other European dissertations

1982present

1861present

(continued)
161

162 TABLE 1
Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

(continued)
Producer or Content Provider
Educational Resource Information Center

Database Name

Content/ Kind of Information

How to Access

Thesaurus With Standardized Keywords?

Yes

Standardized Natural Language Language Searching for Searching Qualitative and Boolean Research? Operators?
Yes Yes

Special Features
Includes curricular materials such as courses of study, books, conference papers, and theses

Dates of Coverage
1966present

Educational Database of Silverplatter Resource journal and Information report literature Center (ERIC) in education and related disciplines MEDLINE Database of OVID references and abstracts covering the fields of medicine, nursing, dentistry, veterinary medicine, the health care system, and the preclinical sciences PsycInfo Index to the Silverplatter research and clinical literature in psychology, psychiatry, and related disciplines

National Library of Medicine (part of NIH)

Yes

No

Yes

International cover1966age; includes present approximately 4,600 current biomedical journals published in the United States and 70 foreign countries

American Psychological Association

Yes

Nounless Yes author lists as key phrase (not a controlled descriptor)

International cover1887age; includes present scholarly journals, dissertations, books, and book chapters

Public Affairs Bibliographic Silverplatter Information index to the Service (PAIS) literature on public policy, social policy, and the social sciences in general PubMed Database of Web at references and Pubmed. abstracts covergov ing the fields of medicine, nursing, dentistry, veterinary medicine, the health care system, and the preclinical sciences Social Science Abstracts (SocSci Abstracts)

Public Affairs Yes Information Service, Inc.

No

Yes

International cover1972age; includes jourpresent nals, books, government documents, statistical compilations, and committee reports International cover1966age; includes present approximately 4,600 current biomedical journals published in the United States and 70 foreign countries; PubMed is the MEDLINE file plus some additional citations to life sciences journals More than 350 key 1983international, Engpresent lish-language periodicals in the social sciences

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

National Library of Medicine (part of NIH)

Yes

No

Yes

Social Work Abstracts

Database of refer- OCLC H. W. Wilson No ences and FirstSearch Company abstracts covering anthropology, economics, geography, law, political science, psychology, and sociology Index to the litera- Silverplatter National Associa- Yes ture in social tion of Social work and Workers, Inc. related disciplines

No

Yes

Yes

Yes

International coverage; includes journals, dissertations, and other materials

1977present

(continued)
163

164 TABLE 1
Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

(continued)
Producer or Content Provider Thesaurus With Standardized Keywords? Standardized Natural Language Language Searching for Searching Qualitative and Boolean Research? Operators?
Yesqualitative methods Yes

Database Name
Sociological Abstracts (Sociofile)

Content/ Kind of Information

How to Access

Special Features

Dates of Coverage

Womens Studies

Database of Silverplatter Cambridge Scien- Yes theoretical and tific Abstracts applied studies covering sociology and related disciplines Interdisciplinary CD-ROM Gale Group No database index(CD edition ing approxiof the mately 100 EngWomens lish language Studies journals Index)

No

Yes

International cover1963age; includes more present than 2,500 journals, meeting abstracts, book reviews, and dissertations International cover1989age; includes scholpresent arly articles, book reviews, and popular materials

Note: NIH = National Institutes of Health.

March 2003, Vol. 25, No. 2

165

TABLE 2: Search Yields by Bibliographic Database

Bibliographic Database
Academic Search Elite AIDS Information Online Anthropological Index Online Anthropological Literature Black Studies Cumulative Index to Nursing and Allied Health Literature Dissertation Abstracts Index Educational Resource Information Center MEDLINE PsycINFO Public Affairs Information Service PubMed Social Science Abstracts Sociofile Social Work Abstracts Womens Studies Total

Citations Identified
63 338 98 338 558 504 181 54 1,101 452 5 1,514 46 285 41 201 5,829

Citations for Inclusion

3 27 1 1 0 38 9 0 28 42 0 31 6 14 4 9 213

% Yield
4.76 6.96 1.02 0.30 0.00 7.54 4.97 0.00 2.54 9.29 0.00 2.05 13.04 4.91 9.76 4.48 3.66

our searches were not high in precision because we intentionally traded precision to heighten recall to ensure an exhaustive search. Working with and around bibliographic databases. Bibliographic databases are not stable entities yielding fixed search results. They may exist one day as independent entities and the next day as inaccessible, extinct, or incorporated into another database. For example, in our library system, access to CD-ROM-based PsycLit was removed soon after we began our searches, leaving us access to an online database, PsycINFO. Journal articles that could be found in AIDSLINE, once an independent database, were incorporated into PubMed as of September of 2001. This change meant that searches on our topic, qualitative studies on women with HIV infection, resulted in inexact recall because PubMed now indexes other health-related journals, in addition to those formerly found in AIDSLINE, and that are searched simultaneously. Databases are updated at different intervals; some are updated daily, whereas others are updated only quarterly or even yearly, thereby contributing to a citation list that may not be current. The searcher is also not a stable, unchanging entity; variations in searchers activities and fatigue or

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

166

Western Journal of Nursing Research

frustration will influence the quality and amount of time spent conducting a search and search results. One of our first challenges was to improve our skills in interacting with these database systems and to work with them to extract the information we were seeking. Each database system offered numerous options for searching, and we had to spend the time investigating which of them would have the best yield in terms of recall. Each database access provider has an information page with its parameters for updates and inclusion dates. We had to become more familiar with the features of each database system, including thesauri and the use of special options, such as Boolean and proximity operators. Almost everyone who has searched a database is familiar with Boolean operators, in which the words and, or, and not are placed between search terms to specify their desired relationship to each other. Proximity operators, used to further customize a search, can request that terms appear adjacent to each other, in a specific order, near each other, separated by a given number of intervening words, or that the terms appear in the same fields (e.g., title field, abstract field). Two examples of the complexity of searching options that we had to master were that PsycINFO has a thesaurus or controlled vocabulary, which is further divided into detailed subheadings used to search for more specific information. MEDLINE uses MeSH terms, as well as less specific subject subheadings; these headings are the keys that unlock the medical literature (Coletti & Bleich, 2001). Decisions thus had to be made regarding which terms to use and how to best use them. For instance, in OVID-based systems, a searcher can explode a search term, which allows the searcher to broaden the search and to further explore the subject matter and related subjects, or a searcher can focus a search term (to narrow the search) to look for citations in which the search term is a major point of the article. We chose to explode our searches, thus maximizing the use of search terms and MeSH headings. A searcher can use Boolean or other operators in these and other databases as well. Boolean operators allow the searcher to use set theory, which is commonly recognized in the form of a Venn diagram of overlapping circles that indicate the and, or, and not functions, to help define the items that will be retrieved by a search. The use of special operators (e.g., proximity operators) is generally available on all of the databases, but some of the databases allow more flexible use than others. Expert-level searching generally entails searches that are conducted by professional search intermediaries who employ appropriate syntax, appropriate use of Boolean and other operators, and inclusion of desired delimiters or expanders for the search.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

167

There were also variations within and among the databases in the mapping of the search terms. In mapping, the system offers additional terminology from the controlled vocabulary of a database in response to a search term entered by the user (Jasco, 1996). Mapping is a relatively recent addition to the usual line of searching features. Within each search, the search terms may be combined with the options selected from the main or basic database search screen, or terms may be entered into the advanced search window, where more options are offered (e.g., publication year, language). These are combined within the system to form what is known as the search string. The system translates the information into the underlying computer query language in operation, maps the information to the appropriate areas, searches for citations satisfying the criteria, and returns the information to the screen. Although there were commonalities among many of the databases, we learned that the vocabulary that was successful in searching for studies in one database was not necessarily successful in searching in another database. Where a database resides can influence searching and subsequent results. The same database provided by two different vendors could conceivably return different results of the same search because searching features vary among database vendors. Minor idiosyncrasies, such as whether a comma should be used between an authors last name and first name (or initial) vary substantially among databases and can be irritating to the searcher, in addition to providing an invalid search result. Few database systems return a message telling searchers that they have made a mistake in entering search terms that a systems language cannot map appropriately. A no records found message may not reflect the absence of records so much as a systems inability to map the search terms used. If the search term is mapped to the best judgment of the programming within the database, totally irrelevant records may appear. The searcher may thus be faced with no citations or many irrelevant ones. Although all the databases we used had characteristics that were supportive of our searches, we found OVID to be the most useful access provider/ vendor for several reasons. There are options in OVID that enable the searcher to see the hierarchical mappings and relationships of individual search terms and to select or deselect the terms. Any search term can be expanded or focused in a variety of ways, and the related mappings in that process can be viewed. Once visualized, we had some control in the selection/ exclusion of a particular mapping. Merging of several individual search results to form a comprehensive search for the phenomenon of interest is relatively friendly in OVID; this helped to reduce the number of irrelevant

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

168

Western Journal of Nursing Research

citations that we had to sift through to find relevant works. OVID provides a seemingly endless number of related subject headings and keywords that represent the indexing of each citation and provide the searcher with a rich source of terms to use. We found this aspect particularly helpful when developing an appropriate search term list. OVID provides a clear error message when a searcher has made an error in handling search terms or applications. OVID also permits the complex use of nested Boolean operators and allows the user to save and e-mail the results in a form that is friendly to importing directly into a reference manager database such as ProCite or EndNotes. Of the 71 published articles and unpublished theses and dissertations that met our search criteria and that we have retrieved thus far, CINAHL and PsycINFO yielded the largest number of citations (43 each), followed by PubMed and AIDSLine, which each had the same 38 citations. There was considerable overlap among these databases. For example, one article (Dunbar, Mueller, Medina, & Wolf, 1998) was cited in 12 different databases. Of the 71 studies, 32 could also be found through the more laborious footnote chasing discussed above. Most of the overlap or duplicate search results may be attributed to inclusion practices of the various indexing and abstracting database producers. The nature of a research project will determine the nature of the search process. Because one of the major objectives in our method project was to explore the various processes involved in conducting qualitative metasyntheses, our search process was necessarily more laborious. Our primary obligation in this project is to do the work required that would put us in a credible position to advise other researchers of work they need not do. In the case of search and retrieval, we deliberately overworked to test how much work was actually necessary to produce a valid search result. Databases to find databases. Knowing where published works likely to be relevant to a project are indexed is critical to finding those works. We used two means to track down citations and to develop our list of databases to search for the project. The publicly accessible Jointly Administered Knowledge Environment (jake at http://jake.med.yale.edu/) database at Yale University (hosted by the Cushing/Whitney Medical Library at the Yale University School of Medicine) improved our ability to identify the electronic index or indices in which a particular journal could be found. We also utilized direct links to some full text articles that are provided in jake. An additional source of information regarding journal indexing is Ulrichs Periodicals Directory, which is a paper-based directory available in most libraries that can also be accessed electronically by subscription.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

169

Both have cross-referencing of journals. But Ulrichs Periodicals Directory (2000) contains many more journal titles and is considered a more comprehensive compilation of journal indexing than jake (Chudnov, Crooker, & Parker, 2000; L. Collins, personal communication, March 22, 2001). Ulrichs also has information about the peer-reviewed status of indexed journals. Developing a list of search terms for the meta-synthesis. The development of search terms appropriate to the area of interest is essential for the effective use of bibliographic databases. Some databases can only be searched by a controlled vocabulary (or thesaurus or list of available terms), others work by natural language processing, which allows common or noncontrolled terms to be searched. Still other databases are searchable using both methods. A controlled vocabulary database means that the effective search terms in the database must be on the list within the database. This list, however, is not always readily apparent. In contrast, natural language processing allows for the searcher to type in virtually any term or phrase, and the database will attempt application. Most databases allow the searcher to select the location to which the search terms, words, or phrases in the citation should be applied: for example, the title, author, abstract, keywords, or words anywhere (meaning every textual piece related to a citation that is in the database). We used the words anywhere option to ensure a comprehensive search, which is a common technique in information retrieval (Coletti & Bleich, 2001; Marchionini, 1995; Meadow et al., 2000). By utilizing words anywhere in our searches, we knew we would capture many more citations than would be relevant, but this gave us more confidence that we would capture more of the relevant literature. We initially thought the search term list would entail a relatively small number of terms, including women, females, mothers, HIV, AIDS, qualitative research, naturalistic research, grounded theory, phenomenology, ethnography, and interview. After a few trial searches, however, we realized that these search terms had neither the specificity nor sensitivity we desired; that is, there were a large number of irrelevant citations produced, although relevant citations that we already knew of were not captured. We had to become more knowledgeable about the underlying mapping patterns for each database to manipulate search terms appropriately. For example, in some databases (e.g., PsycINFO), the search term women maps to human female. Human female includes mothers, sisters, widows, daughters, and wives, as well as battered females, working women, and female criminals.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

170

Western Journal of Nursing Research

The database may be searching for key words as provided by the author, the indexer, or contained in the title, abstract, or in the full text of the article. In OVID-based MEDLINE and CINAHL, these mappings can be selected or deselected, depending on the focus of the search and retrieval activity. We therefore manipulated our search terms according to the parameters of the systems we searched. Qualitative research. An especially challenging problem was the difficulty locating exclusively qualitative research reports. As a search term, qualitative research yielded few relevant citations and many irrelevant ones, such as qualitative laboratory assays. We accordingly generated a more refined list of qualitative research search terms. The challenge was to use search terms and operators that spoke the language of each of the bibliographic databases while still representing qualitative research. Qualitative research in MEDLINE maps to a large number of subject headings: research, research design, nursing methodology research, nursing research, data interpretation, health services research, nursing, and myocardial infarction. Following hierarchical mapping, each of these subheadings then maps to other terms. In OVID-based CINAHL, qualitative research maps to qualitative studies. As stated earlier, we also searched using the many terms that indicated that the methodology used was qualitative in nature, and we truncated the terms as well to cast a wider net. The most important thing we realized, particularly because we are doing qualitative meta-synthesis work, is that most bibliographic databases do not index articles according to research methodology. Table 1 lists which of the 16 databases we used indexes according to research methodology. Determining relevance. Even when we refined our search terms and techniques, using the berrypicking process, there were still hundreds (about 20%) of citations retrieved that could not be clearly excluded on the basis of title and abstract alone. We therefore developed a process to track the decisions we made about these citations. This process, which deals with both substantive and methodological issues, is graphically shown in Figure 1. Once a citation was obtained, we were sometimes able to exclude it based on the title alone. If we were unsure, we checked the abstract. A citation could then be excluded at this point as not meeting the criteria, or it could merit further investigation. If further investigation was warranted, the full article was obtained. At that point, the citation was included, excluded, or its status remained uncertain. Uncertain citations then led to a negotiation of consensus on its status and further refinement and delineation of inclusion criteria.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

171

Search Term List

Search Citation Database

Check Title

Include

Unsure

Exclude

Check Abstract

Include

Unsure

Exclude

Check Full Article

Include

Unsure

Exclude

Consult for Consensus

Include

Exclude

Inclusion for Bibliographic Sample

Exclusion from Bibliographic Sample

Check Bibliography

Check Author Citations

Initial/Ending Activity Decision Point Specific Step, w/ or w/o citation document Specific Step, w/ citation document (electronic/paper) Study Electronic Management Database Descendency Ancestry

Figure 1. Search, retrieval, and validation process. NOTE: Reference for graphic symbols drawn from Harris (1999, p. 156).

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

172

Western Journal of Nursing Research

Once any article was finally included as part of the bibliographic sample for the meta-synthesis, its reference list was checked for other studies that might meet our criteria, and the author was searched in all of the different databases to see if he/she had published something else that met the criteria, both strategies described above in the berrypicking techniques. As shown in Figure 1, a full text copy was appraised for every study that was included in the meta-synthesis to ensure it met all study criteria. This often required obtaining copies of articles through interlibrary loan for a peritem fee. Although we ultimately excluded many of these articles, we considered this neither a waste of time nor money as we gained articles that will be useful to us in contextualizing the results of our meta-synthesis. Searching for and retrieving unpublished theses and dissertations presented a special challenge to determining relevance. We located most of the theses and dissertations we were interested in through Dissertation Abstracts International (DAI), although some of them appeared in other databases as well, such as PsycINFO or Sociofile. Our initial search of DAI yielded 8 theses and 181 dissertations. The search for unpublished works can be a costly venture that some searchers may not be able to afford because most of these works would have to be purchased to determine their relevance to a project. In most cases, these works could not be obtained on interlibrary loan. If the researchers cannot afford to purchase these works, this may contribute to a less complete bibliographic sample for meta-synthesis.

MANAGING INFORMATION

Projects such as this generate volumes of information that require efficient management and cataloging. The foundation for ensuring valid procedures and results is the establishment of a clear audit trail documenting all procedural moves and decision-making points. For example, we had to find a way to track citations that were removed immediately, that were ultimately accepted, and that required more investigation. All the database systems we used supported the transfer of our search results to a printer and/or to saving them on a disk or hard drive. With the exception of Black Studies, Womens Studies, Anthropology Literature, Anthropology Index, and PAIS, all the database systems we used supported the transfer of our results through email. OVID-provided searches are also readily transferable into ProCite or other reference manager software packages, making management of the search results highly efficient and effective. Reference manager software, such as ProCite and EndNote, proved invaluable to us in managing the

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

173

results of our searches. We created several Access databases for managing those citations that required more extensive tracking and handling, such as Dissertation Abstracts. Information can be readily transferred between EndNote, ProCite, and Access through several mechanisms. The publicly available PubMed system is directly searchable, and our search results were directly downloadable through our universitys ProCite license.

OTHER THREATS TO VALID SEARCHES

Despite the steps we have taken to ensure a valid search, there are other threats to validity over which a searcher has no control. For example, most database systems (e.g., MEDLINE ) have human indexers, and as with any work performed by humans, they can make mistakes entering information into an electronic citation database (Coletti & Bleich, 2001; Hjorland, 2001; Meadow et al., 2000). Other types of errors, from authors and editors, can affect the validity of a search as well. In a study that examined the accuracy of entries in MEDLINE by searching for misspelled text words, a total of 200 citations were retrieved from 10 selected search terms. The authors found that most (141/200; 71%) misspelled words occurred within the abstract only and that 98 of the 200 articles with misspelled text words might be missed if a searcher conducted a MEDLINE search using the correctly spelled word alone without the heading (Ray & Vermeulen, 1996).

CONCLUSIONS AND RECOMMENDATIONS

We have tried here to fix in time and space a process that is in reality a moving target. Acknowledging the situational dynamic of searching for and retrieving qualitative studies, we offer recommendations for searchers and writers of qualitative reports that we hope will help them manage it. We have a suggestion for indexers, however, based on Evanss (2002) assertion that the effectiveness of electronic database searchers would be improved through the development of a qualitative research search filter. This type of methodological filter is a predetermined search strategy that uses terms related to research design to identify all those studies using the research method of interest to the searcher. The purpose of such a filter would be to retrieve only those studies most relevant to the review (in this case, those using qualitative methods) (Evans, 2002).

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

174

Western Journal of Nursing Research

Recommendations for Searchers

Because there is a different calculus involved with every search, searchers must figure out the specific calculus for their particular project or topic. This involves configuring a number of factors. Searchers must determine the resources they will have available to them for their search, including the amount of time they can expend on locating relevant works, the amount of money they can spend to retrieve full-text copies of works, and the richness of their search environment, including the availability of expert librarians, servers, and citation databases. This portion of the calculus will help searchers define their overall search strategy: that is, how to best approach the search process and whether to emphasize precision or recall. Because our research project demanded that we investigate and then recommend the best procedures to conduct qualitative meta-synthesis studies, we deliberately chose more laborious and time-consuming search approaches to determine whether less laborious or time-consuming methods would produce the same results. For example, we reviewed more than 3,600 citations appearing in the reference lists of reports we accepted into our bibliographic sample, in addition to those in other works about women with HIV infection. This yielded 32 studies that were ultimately included in our sample. Yet, of these 32 studies, all could be found electronically as well. We have located 1 study that appeared as a chapter in a book (Kass & Faden, 1996) and was not available as a citation on any of the databases we searched. We will not know until we are further into the meta-synthesis process itself if we will find any other studies exclusively through a hand search of a books chapters or the reference lists in a book or in articles. Of greater importance are the consequences of not finding studies by doing electronic searches alone. Would the studies that a meta-synthesist does not find change the results? We might find that, in our case, the time we spent in doing this ancestry work was unnecessary. Because qualitative research spans disciplines in the sciences, humanities, and arts, searchers must also determine which databases in these varied disciplines they must have access to so as to achieve a valid search. This in turn requires searchers to learn the intricacies and idiosyncrasies of working with and around these databases. Searchers may need to spend more time than they had anticipated enhancing their search and retrieval skills, which can be done with tutorials available on most bibliographic databases. We have seen, in hindsight, the need for an information specialist on a research project such as ours. The irony is that with so much information available at our fingertips, retrieving relevant materials has become harder, in part, because of advances in technology, but also because of the lack of standard-

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

175

ization across database systems. Every bibliographic database has its own unique features, and even these features can change with some frequency. Searching in and of itself is an area of specialized knowledge, and metasynthesists would be advised to have an information specialist as part of the team. Searchers will also need to have the flexibility to match the ever-changing situational dynamic of the search process itself. They will likely have to rethink and refine their inclusion and exclusion criteria as the search begins to yield studies. Because there is no fixed endpoint to searching, searchers must decide at what point they will be satisfied with their results and, more important, at what point they can stop searching and still meet the criterion for having conducted an exhaustive search that all research integration projects require. Because there is a lag time between the arbitrary end of a search and the completion of a meta-synthesis of the findings from the reports retrieved from that search, researchers must decide whether they will update their search before submitting their work for publication.
Recommendations for Writers

Although we have emphasized searching and searchers here, we found that writers play a key role in determining whether their work will be found. Evans (2002), in discussing the importance of systematic reviews for evidence-based practice, noted that although methodological search filters have worked well with locating randomized controlled trials, the unique characteristics of qualitative research limit the easy translation of these search techniques. He noted that the lack of a clear description of the method used in the title or abstract of a study makes it difficult to find qualitative studies when doing electronic searches. Although constrained by publication venues and database systems, writers could contribute to the retrievability of their work and to research dissemination and utilization (e.g., Barry, 1998; Coletti & Bleich, 2001) if they attended more thoughtfully to those aspects of research reports that are most visible in electronic systems, including keywords and the construction of the title and abstract. For example, we recommend that, whenever possible, researchers use the term qualitative research as a keyword in every report of a study using any qualitative methodologies. Although more challenging to achieve, given the regular use of established thesauri and subheadings, we would also recommend that indexers use the same term to index any study conducted using any qualitative methodology. Searchers could then search under that term rather than having to search with terms covering every kind

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

176

Western Journal of Nursing Research

of qualitative method or technique. Because one of the goals of research is to disseminate the findings to an audience, writers need to do everything they can to ensure that they are providing appropriate terms to aid searchers. Writers should also be aware that searchers have a heightened expectancy with bibliographic databases. Searchers become impatient while waiting even a few seconds because of the fundamental change in temporal expectations fostered by computers. Electronic environments raise searchers expectations for information access. But these expectations interact with searchers physiology and psychology. Searchers must still perceive, process, and comprehend information if they are to achieve their goals (Marchionini, 1995). Writers need to know this to ensure that what is relevant about their work will be as immediately accessible to the searcher as possible. Searchers make critical decisions about whether to investigate citations further based on what is in front of them on a computer screen. If a citation appears immediately to be irrelevant, searchers may immediately exclude it from consideration, and the cycle of research, research dissemination, and research utilization may also, therefore, be halted. In summary, although there is still much debate on what qualitative metasynthesis projects entail and how they should be conducted, there is no argument that any meta-synthesis of qualitative findings rests on a sufficiently exhaustive search. Qualitative research presents both searchers and writers of qualitative reports many challenges that must be met to make the best and the most of the qualitative findings that offer so much to the health and wellbeing of the people nurses serve.

NOTES
1. The authors acknowledge the assistance of Jean C. Blackwell and Lynne D. Morris, Health Sciences Library, the University of North Carolina at Chapel Hill. 2. This study, titled Analytic Techniques for Qualitative Metasynthesis, is supported by grant no. R01 NR04907 from the National Institute of Nursing Research.

REFERENCES
Barry, C. L. (1998). Document representations and clues to document relevance. Journal of the American Society for Information Science, 49, 1293-1303. Bates, M. J. (1989). The design of browsing and berrypicking techniques for online search interface. Online Review, 13, 407-424.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

March 2003, Vol. 25, No. 2

177

Campbell, R., Pound, P., Pope, C., Britten, N., Pill, R., Morgan, M., et al. (in press). Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine. Chudnov, D., Crooker, C., & Parker, K. (2000). Jake: Overview and status report. Serials Review, 26(4), 12-17. Coletti, M. H., & Bleich, H. L. (2001). Medical subject headings used to search biomedical literature. Journal of the American Medical Informatics Association, 8, 317-323. Cooper, H. (1998). Synthesizing research. Thousand Oaks, CA: Sage. Death at the hands of science. (2001, July 31). Retrieved from www.nytimes.com Dixon-Woods, M., Fitzpatrick, R., & Roberts, K. (2001). Including qualitative research in systematic reviews: Opportunities and problems. Journal of Evaluation in Clinical Practice, 7, 125-133. Dunbar, H. T., Mueller, C. W., Medina, C., & Wolf, T. (1998). Psychological and spiritual growth in women living with HIV. Social Work: Journal of the National Association of Social Workers, 43(2), 144-154. Estabrooks, C. A., Field, P. A., & Morse, J. M. (1994). Aggregating qualitative findings: An approach to theory development. Qualitative Health Research, 4, 503-511. Evans, D. (2002). Database searches for qualitative research. Journal of the Medical Librarians Association, 90, 290-293. Glass, G. V., McGraw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage. Harris, H. (1999). Information graphics: A comprehensive illustrated reference. New York: Oxford University Press. Haynes, R. B., Wilczynski, N., McKibbon, A., Walker, C. J., & Sinclair, J. C. (1994). Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association, 1(6), 447-458. Hjorland, B. (2001). Toward a theory of aboutness, subject topicality, theme, domain, field content . . . and relevance. Journal of the American Society for Information Science and Technology, 52, 774-778. Jasco, P. (1996). Ovid web gateway: Nobody does it better. Online, 20(6), 24-30. Jensen, L. A., & Allen, M. N. (1996). Meta-synthesis of qualitative findings. Qualitative Health Research, 6, 553-560. Kass, N., & Faden, R. (1996). In womens words: The values and lived experiences of HIVinfected women. In R. R. Faden & N. E. Kass (Eds.), HIV, AIDS and childbearing: Public policy, private lives (pp. 426-446). New York: Oxford University Press. Kearney, M. H. (1998). Ready-to-wear: Discovering grounded formal theory. Research in Nursing & Health, 21, 179-186. Losee, R. M. (2000). When information retrieval measures agree about the relative quality of document rankings. Journal of the American Society for Information Science, 51, 834-840. Marchionini, G. (1995). Information seeking in electronic environments. New York: Cambridge University Press. Mason, J. (1996). Qualitative researching. London: Sage. Meadow, C. T., Boyce, B. R., & Kraft, D. H. (2000). Text information retrieval systems. San Diego, CA: Academic Press. Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography synthesizing qualitative studies. Beverly Hills, CA: Sage. Ray, J. G., & Vermeulen, M. J. (1996). Words, words, words: Mizspellin and MEDLINE. British Medical Journal, 313, 1658-1659.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009

178

Western Journal of Nursing Research

Sandelowski, M., Docherty, S., & Emden, C. (1997). Qualitative metasynthesis: Issues and techniques. Research in Nursing and Health, 20, 365-371. Schrieber, R., Crooks, D., & Stern, P. N. (1997). Qualitative meta-analysis. In J. M. Morse (Ed.), Completing a qualitative project (pp. 311-326). Thousand Oaks, CA: Sage. Sutcliffe, A. G., Ennis, M., & Watkinson, S. J. (2000). Empirical studies of end-user searching. Journal of the American Society for Information Science, 51, 1211-1231. Ulrichs Periodicals Directory. (2000). New York: Bowker. Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.

Downloaded from http://wjn.sagepub.com at UNIV OF TEXAS AUSTIN on November 17, 2009