PRO development: rigorous qualitative research as the crucial foundation

Author(s): Kathryn Eilene Lasch, Patrick Marquis, Marc Vigneux, Linda Abetz, Benoit Arnould,
Martha Bayliss, Bruce Crawford and Kathleen Rosa
Source: Quality of Life Research, Vol. 19, No. 8 (October 2010), pp. 1087-1096
Published by: Springer
Stable URL: http://www.jstor.org/stable/40927744
Accessed: 27-04-2015 18:44 UTC
REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.org/stable/40927744?seq=1&cid=pdf-reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact support@jstor.org.

Springer is collaborating with JSTOR to digitize, preserve and extend access to Quality of Life Research.

http://www.jstor.org

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

Qual Life Res (2010) 19:1087-1096

DOI 10. 1007/s11136-010-9677-6

PRO development:rigorousqualitativeresearchas the crucial

foundation
KathrynEilene Lasch *PatrickMarquis *
Marc Vigneux* Linda Abetz * BenoitArnould*
Martha Bayliss Bruce Crawford* KathleenRosa

Accepted:6 May 2010/Publishedonline: 30 May 2010

The Author(s)2010. This articleis publishedwithopen access at Springerlink.com

Qualitativeresearch
Abstract Recentlypublishedarticleshavedescribedcri- Keywords PRO development
teriato assess qualitativeresearchin the healthfieldin Groundedtheorymethods
general,butveryfewarticleshave delineatedqualitative
methods
tobe usedinthedevelopment
ofPatient-Reported
Outcomes(PROs). In fact,how PROs are developedwith Qualitativeresearchin PRO development
focusgroupsand interviews
has been
subjectinputthrough
when Patient-reported
outcomes(PROs) in clinicaltrials,effecgiven relativelyshortshriftin the PRO literature
articleson the tivenessstudies,and public health researchhave been
comparedto the plethoraof quantitative
of PROs. If documentedat all, denned as "any reportcoming directlyfromsubjects
psychometric
properties
mostPRO validationarticlesgive littleforthe readerto without
of thephysicianor othersabouthow
interpretation
evaluatethecontentvalidityof themeasuresand thecred- theyfunction
overallor feelin relationto a conditionand
of themethodsused to develop itstherapy"[1, p. 125J.The valueofqualitativeresearchin
ibilityand trustworthiness
them.Increasingly,
scientists
andauthorities
want thedevelopment
of PRO measureshas beenrecognizedfor
however,
to be assuredthatPRO itemsand scales havemeaningand many years. Witnessthe growingacceptanceof such
relevanceto subjects.This articlewas developedby an
researchby a new editionof a book thatdevoteda brief
international,
interdisciplinary
group of psychologists, chapterto qualitativeresearchin an otherwisecomprehensivevolumeon quantitative
methodsthatare used to
psychometricians,
regulatory
experts,a physician,and a
It
and
measure
of
life
in
clinical
trials[2]. A morerecent
sociologist. presents
rigorous appropriate
qualitative
quality
researchmethodsfor developingPROs with content focushas beenplacedon theconceptsbeingmeasuredand
- notin termsof correlation
The approachdescribedcombinesan overarching theirmeaning
coefficients
or
validity.
theoreticalframework
with grounded factorialstructure,
buttheirauthenticity
forsubjects,i.e.,
phenomenological
data collectionand analysismethodsto yieldPRO
theircontentvalidity.
The emergence
ofcontentvalidity
as
theory
itemsand scales thathavecontentvalidity.
a construct
was to guardagainststrictly
numericalevaluationof testsand othermeasuresthatoverlookedserious
threatsto the validityof inferencesderivedfromtheir
scores

[3]. This articlepresentsan approachincorporating

K. E. Lasch (El) P. Marquis M. Bayliss K. Rosa
USA
an
Values,
MA,
Boston,
Mapi
over-arching
phenomenological
approachintogrounded
e-mail: kathy.laschi^mapivalues.com
theorydata collectionand analysismethodsto mostaccuratelyincludethesubject'svoice in PRO development.
M. Vigneux B. Arnould
The quest forauthenticity
in instrument
France
Values,
development
Mapi
Lyon,
evolvedfroma pragmatic
approachrangingfromliterature
L. Abetz
review,clinicianexpertise,and the psychometric
perforMapi Values, Bollington,UK
mance of itemsfromlarge samples and batteries(e.g.,
Medical OutcomesStudyShortForm [SF-36]) to direct
B. Crawford
involvement
Mapi Values, Tokyo,Japan
by subjects in item generation14]. When
Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1088

Qual Life Res (2010) 19:1087-1096

subjectshave been includedto date, however,the systematicanalysisof theirwords and the link fromtheir
words to conceptsunderlyingitems is usually neither
documented
nortransparent.
and systematiTransparency
zation,however,are consideredhallmarksof good qualitativeresearch[5]. Theirabsencein qualitative
researchin
the PRO field makes it difficultto communicateand
compareresults.Otheressentialissues in theconductof
rigorous qualitative research for PRO development
include:whodoes oneinterview,
howdoes one analyzethe
data systematically
and transparently,
how does one
to undergird
a questiondevelopa conceptualframework
nairefromparticipants'
responses,and, above all, what
theoretical
framework
(a guide as to which
overarching
and
which
between
thoseconcepts
concepts
relationships
shouldbe thefocusofa researchstudy),ifany,wouldbest
serve PRO development?A conceptualframework
as
definedby theFood and DrugAdministration
(FDA), one
of the majorconstituents
forPROs, represents
the demonstratedrelationships
betweenand among items on a
and domains(multidimensional
conceptin
questionnaire
whichitemsare groupedtogether)!
1, 6].
in2006
TheFDA issueda PRO draft
guidancedocument
in 2009 that,whenfoland a finalGuidanceto Industry
lowed, makes it criticalfor instrument
developersor
methodsin
to use andunderstand
state-of-the-art
reviewers
qualitativeresearch[6-8]. Adherenceto thisguidelineis
is intendedas an endpointto
requiredif thequestionnaire
benefitassessingclear conceptsthat
evaluatetreatment
claim.Recently,
a labelingand/or
advertising
might
support
membersof the StudyEndpoints& Label Development
atthe45th
(SEALD) divisionintheFDA gavepresentations
Associationin
AnnualMeetingof theDrug Information
of contentvalidity
whichtheyemphasizedtheimportance
and qualitativemeasure.Contentvalidity,
as an important
in general,meansthata measurecaptureswhatit intended
theFDA morespecifito measure.In thesepresentations,
of
a
PRO
as (1) evidencethat
content
defined
validity
cally
theitemsand domainsmeasuretheintendedconcepts,as
anddesiredclaim;(2)
framework
depictedintheconceptual
evidence that the items,domains,and concepts were
compredevelopedwithsubjectinputand are appropriate,
the
and
that
hensive,andinterpretable
(3)
study
bysubjects;
of
the
is
targetpopulation.
sample representative
Boththecollectionof qualitativedata and its analysis
and rigorousin thepast
have becomemoresystematized
have increasingly
30 yearsas healthresearchers
incorporatedthemintotheirwork.The mostinformative
waysto
Even whenprohave been refined.
interview
participants
to conductfocus
videdwithdiscussionguidesand training
groups or in-depthinterviews,however,interviewers
in qualitative
researchmethodsuse theseguides
untrained
interview.
as thoughtheywere conductinga structured

They oftenask questionsthatput wordsin the subjects'

mouthsand do notdig deeperthanwhatis directly
asked
the
of
Probesthat
(or rarelygo beyond scope questioning).
ask a studyparticipant
to describemorefullythemeaning
of a conceptthatis spontaneously
offered
are rarelyused.
Using guidesas rigidscriptslimitsthecollectionof data
thatis ideal forcapturing
subjects'meaningof theexperienceofa condition
anditstreatment.
In addition,
thePRO
fieldgenerallyhas nottakenfulladvantageof thedecades
of knowledgein thefieldof surveyresearchpsychology
to
construct
itemsand responsesthatmostclearlydepictthe
or an impactof a treatment
or a
experienceof a symptom
condition[9-11].
Researchers
havepublishedorpresented
criteria
on how
to evaluatequalitativeresearchin healthliterature
in generaland in thedevelopment
ofPROs in particular
[12-16].
is availablein thePRO
However,verylittleinformation
fieldon how to collectand analyzequalitativedata comon psychometric
methods
paredto theplethoraofliterature
to supportthe validityof PROs. Only one articleto our
knowledge,published in 2008, specificallydiscusses
qualitativeresearchmethodsto assureclarityand content
validityin PROs [17].
We presentan approachto developa PRO instrument
withcontentvalidity.This approachwas developedby
an international,
team of psychologists,
interdisciplinary
psychometricians,
regulatory
experts,a physician,and a
sociologistwithover 25 yearsof experienceconducting
qualitativeresearch.We describehow qualitativeresearch
andthepsychology
ofsurveyresponsemaybestbe applied
to subto captureboththemeaningof medicalconditions
jects and treatment
impact.
Briefbackground:psychologyof surveyresearch
and qualitativehealthresearch
researchis
Similarto itsquantitative
equivalent,
qualitative
modelsand data
an umbrellatermforvarioustheoretical
collectionmethods[18, 19]. Anthropologists,
sociologists,
nursingresearchers,and, recently,psychologistshave
to thehealtharena[7,
appliedvariousmethodsandtheories
on thepsychology
20-23]. Thereis also extensiveliterature
answer
of surveyresearchthataddresseshow respondents
itemson a questionnaire
[9-11, 24-28]. The mostcommodel
monlyused cognitivemodelis thequestion/answer
This
model
idenin
1984
[29].
proposedby Tourrangeau
tifiesthecognitivestagesin answeringa surveyquestion,
retrieval,judgment,response
includingcomprehension,
takes
selection,andresponsereporting
[25]. Thisliterature
context
intoaccounttheinteractive
aspectsoftheinterview
and thecognitiveprocessesthatare involvedin answering
items. Its focus has been on the improvementof

) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

Qual Life Res (2010) 19:1087-1096

designratherthanthe blankslate involvequestionnaire

mentof subjectsto captureimportant
concepts.
frameThereis no consistentapproachor theoretical
work,however,in this broad-basedresearchthat one
mightuse as a guidelineto applyqualitativeinquiryto
thedevelopment
of PROs [30-33]. Studiesoftenprovide
of verygeneralthemes,butfocusingon
counts
frequency
with
such
a small and variednumberof subfrequency
value of qualitativeresearch.
limits
the
informative
jects
Rarely(if ever)is a conceptualframework
developedthat
could underpinan instrument.
Clinical terms,such as
"cancer-related
fatigue,"are oftenused to portrayor
definea concordancebetweenthe termand subjects'
experiences.However,numerousstudiesexemplified
by
Schwartzand othershave documentedthe discordant
of manysymptoms
betweensubjectsand their
perception
providers[34].
Overviewof approachesin qualitativeresearch
Qualitativeresearchin thehealtharenahas reliedon several approachesto collect and analyze data, as well as
and presentresults.These mainlyincludepheinterpret
nomenology,ethnography,
groundedtheory,discourse
contentanalysis.All approaches
analysis,and traditional
and limitations,
and theyshouldbe chopossessstrengths
sendependent
the
of
research
upon type
question(s)asked.
Table 1 brieflydescribesthe main distinctions
between
thesemajorapproaches;onlyphenomenology
and grounded theory
theapproacheswe use,are morefully
methods,
describedin thisarticle.Table 1 summarizes
theessence,
samplingmethods,data collectionand analysismethod,
andresultsyieldedbycommonly
usedqualitativemethods
in healthscienceresearch.Please notethatresearchers
use
thenamesoftheseapproachesinterchangeably
at timesand
haveslightly
different
of them.
interpretations
both
an
theoretiUsing
overarching
phenomenological
cal framework
andgrounded
theorymethodsappearsmost
suitedto assuringthecontent
validityand meaningofPRO
In addition,it is
conceptsfromthepatient'sperspective.
mostconduciveto developingconceptualframeworks
of
as
the
FDA.
We
chose
these
two
questionnaires required
by
to
approachesbecause theyseemedthemostappropriate
ensurethefollowing:
Using subjects'own wordsto describetheirexperiences ratherthan using broaderthemesdeveloped
theeyesof theresearcher
to describepatients'
through
as
in
traditional
content
experiences,
analysis;
Adequatesample size to ensureachievingsaturation
or setsof
(no newconceptsproducedin finalinterview
not generallya goal of narrowly
defined
interviews);

1089

a case study,discourse
phenomenology,
ethnography,
or
content
but
forPRO
analysis,
analysis
important
contentvalidityand in groundedtheorydatacollection
and analysismethods;and
The abilityto developitemsfora PRO measurerather
than produce a narrativeaccount of a subject's
experienceswithina social context.

Phenomenologicaltheoreticalframework
The scope forphenomenological
researchhas beensimply
definedas "researchdesignedto discoverand understand
the meaningof humanlife experiences"[35, p. 114].
seeks "to understand
thelivedexperience
Phenomenology
of individualsand theirintentions
withintheirlifeworld"
[18,p. 24]. It answersthequestion,"Whatis itliketo have
a certainexperience?"[18, p. 24] Althoughphenomenoseveralqualitativeapproaches,
the
logicalinquiryunderlies
beliefis thatthewayto studya phenomenon
is
underlying
to access it through
theeyes of theperson(s)havingthe
thesine qua non
experience.This makesphenomenology
framework
for
PRO
research.
It is the theooverarching
reticalunderpinning
thatcan guideresearchquestionsin a
discussionguideand data collection,forexample.Thus,a
It is neither
constrained
typicalquestionis open-ended.
by
that
preconceivedtheoriesnor takenfroman instrument
may possess sound psychometricpropertiesbut was
developedwithoutsubjectinput.In a conceptelicitation
forexample,on cancer-related
interview,
fatigueone asks,
"Please tell me about some of the symptomsyou have
fromcanceror cancertreatment?"
rather
than
experienced
a
the
word
asking questionusing
fatigue,unlessthesubhave used it first.One probesforclarifijects themselves
cation on the meaningof responseslike "tiredness,"
aregiven
"weary,""weak,"or "low energy."Participants
ampletimeto expressthemselves.
Groundedtheorymethodsapproach
As thegoal is an analysisthatproducesnotonlyconcepts
butalso a framework
of itemsto be used as endpointsin
clinicaltrials,theanalysisand outputof a purephenomenologicalapproachare insufficient.
Usingtheumbrellaof
we
that
phenomenology, suggest
groundedtheorydata
collectionandanalysismethodsbestservethedevelopment
of a PRO structured
thatcan be used as an
questionnaire
in
a
clinical
trial.
endpoint
Groundedtheoryis morea set of methodsthana real
It can be seen as a "logicallyconsistent
setof data
theory.
collection and analytic proceduresaimed to develop
Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

Qual Life Res (2010) 19:1087-1096

1090
Table 1 Comparisonof qualitativeresearchapproaches

Essence

Phenomenology Ethnography

GroundedTheory

Case Study

To understand
the Immersionof
researcherin
meaningof
settingto
participants'
understand
the
experiences
withintheirown ways of lifeof a
"lifeworld"
culturalor social
group

Set of data collection To yielda full

and analysis
or
description
methodsthatassure explanationof a
thatthemeaningof phenomenon
a conceptis
withina real life
discoveredin the
setting,
e.g., an
wordsand actions
Alzheimer'sunit
of participants
from
thegroundup- not
fromapplicationof
a prioritheoryor
concepts

DiscourseAnalysis

ContentAnalysis

To describehow and Researchercodes and

abstractsinto
whysocial
interactions
are
meaningunits
notes
enacted
observational
routinely
or transcripts
of
usinganalysisof
naturally
occurring interviews,
talkand texts(e.g., avoidingspecific
subject-physician verbatimreports,
Oftenuses prior
interaction)
theoryfrequency
countsto describe
themes
prominent
in text

A case embeddedin Randomsamplingof Observationsor

as
Progressive,
or interviews
a singlesocial
text,encounters,
theoryis built;
numberof
settingbut
samplingof social
interactions
samplingof
participants
events,key actors,
dependson
etc. occurs
saturation;
theoretical
(purposeful
sampling)
sampling
Textualdata from
Observationor
Data
In-depthand/orfocus In-depthinterviews Observations,
In-depth
of
archivaldata,
in
Collection conversations
transcripts
recordingof
groupinterviews; with20-30
with *
clinicalinterviews interviews
interviews
observation
which
participants,
focus
interviewer
participants,
dependson
of
bracketshis/her
groups,or
homogeneity
data
own experiences
published
participants;
documents
collection
fromthoseof
continuesuntil
interviewee
saturation
achieved
and Readingthrough
Data
analyzed Data usuallycoded
Transcripts
PhenomenologicalDescription,analysis, Coding,sorting,
intoabstractcodes
to
data
data"a transcript, withattention
and interpretation integrating
reductionand
Analysis
fromverbatim
of thesocial or
structural
notes,documents, minutiathatmight and developed
the
otherwisebe
culturalgroup;
through
objects;make
report,and
synthesis;
considered
researcher
eyes
interpretative
marginnotesand
inductively
analysismay
of researcher;
forminitialcodes;
identifies
essence proceedin a
"noise," e.g.,
buildinga
words
describecase and
of phenomenon numberof ways
codes, concepts,or
hesitations,
conceptual
themescountedin
to
and clustersdata includingbuilding framework
context;aggregate such as "dunno,"
termsof relative
taxonomiesand
aroundthemes
etc.; data are
categoriesand
explaina
as seen
discover
importance
analyzed
patterns
phenomenon.
making
and
of categories;
by researcher
inductively
comparisons;often Iterativeprocess
eventsand talkare
and
drawsconnections wherebyfurther
interprets
seen as socially
makessense of
data collectionis
betweenthe
constructed
of the
through
findings
by
prompted
description
theinteraction
groupand broader researcher's
extanttheoretical
analytic
uses
frameworks.
interpretation;
constant
comparison
method.Data
collectionstops
of
whensaturation
conceptsachieved.
NarrativeaugmentedDescriptionand
A conceptualor
Results
Frequencycountsof
Descriptionof the Rich narrative
themesand
of
model
theoretical
of
is
tables,
explication
figures,
by
description
phenomenon
actionsin everyday descriptive
and sketches
thatdescribes
oftenpresented culturalor social
quotes
fora code
and institutional
as narrative
conceptsor
group,i.e., story
withcharactersand categoriesand their
settingsthrough
a plot
analysisof talkor
relationships;
speechacts
presented
usually
as a visual graphic

Sampling

Few participants, Key informants;

of
usually<6, who observation
have experienced events;possibly
thephenomenon participant
observation

) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1091

Qual LifeRes (2010) 19:1087-1096

theory"[21, p. 27]. These qualitiesmake it especially

thedimensions
in thePRO fieldwhencapturing
pertinent
from
thedevelopment
ofa conceptand makingtransparent
and
verbatim
conceptsin textualdata to itemgeneration
domains[21, 36, 37]. It helps
of meaningful
development
intoa conceptual
conductandanalyzeinquiry
investigators
the
thatcan thenbe used to testquantitatively
framework
of
a
PRO
instrument.
and
reliability validity
Humanistassumptionsunderliethe use of grounded
in thesensethatitaccountsforan indexof feelings
theory
an expeare seen as representing
or meanings.Interviews
riencethatallowsaccess to authentic
privateselves,gives
voice to thevoiceless,and offsetstheerrorsof positivism
NathanGlazerandAnselm
andprejudice[38]. Itsfounders,
to
intended
research
to be a precursor
Strauss,
qualitative
a
of
research
andwrote clearset
morerigorous
quantitative
guidelines[36].
Accordingto theseguidelines,thereare threeessential
ofqualitative
research:(1) itencourages
keycharacteristics
to
their
or feelingsusingvery
thoughts
participants express
littlestructure
interviews;
(2) it is iterativein the
during
sensethatconceptsfoundin thedata lead to otherinterviewsto lookforidenticalconceptsorclarification
ofthose
of a conconcepts;and (3) itsuse facilitates
development
thansubstantiation
rather
of an a priori
ceptualframework
of a set of concepts.
interpretation
Theoreticalconceptsused whenconductinginterviews
emergefromthedata.Charmaz'snotionsof changedselfof chronically
ill subjectsemergedfromthedata
identity
for
a
withmultiplesclewhen, example, studyparticipant
rosismentioned
to herthatwhenshe was havinga "bad"
and...thattimehad a
day,she dealt"withtimedifferently
different
of
meaningto" her[39,p. 31]. Thisincorporation
avoids theclassic pitfallof
conceptsused by participants
takingfor grantedthatthe researchershares the same
as therespondent
[39]. Charmazfurther
meanings
explored
theconceptofgoodandbad daysandfoundthatgooddays
indicated"minimalintrusiveness
of illness,maximalcontrolover mind,bodyand actions,and greaterchoices of
- all potentiallyimportantconcepts in the
activities"
experienceand impactof chronicillness[39, p. 31].
Grounded
methods
arecharacterized
theory
byinductive
to thegeneral)ratherthandeductive
(fromtheparticular
(fromthe generalto the particular)
reasoning.Data collectionand analysistypicallyproceedin a simultaneous,
iterativefashion.Researcherscreate analyticcodes and
categoriesfromdata, not preconceivedhypothesesto
whichdatamightbe overlaid.The researcher
continuesto
use "memos,"or thoughts
or insightsthattheymayhave
aboutthedataduringanalysis.The term"memos"refers
to
writesduringthecoding
analyticnotesthattheresearcher
thedata,whichserveas reminders
of
processconcerning
and
directions
in
which
further
data
important
thoughts

collectionand analysisshouldgo. Memos are conceptual

and analyticalratherthandescriptive
[37]. The researcher
will searchforcodes,usinga searchfunction
and Boolean
definedby "and," "or," "not"),
operators(relationships
and develop modelsto explainthe relationships
between
coded data. One could,forexample,searchforall quotes
that containedthe IBS codes "abdominal pain" and
"cramping."All quotesthatare givenbothof thesecodes
can be outputto examinethemeaningof thesepain conof theother?
ceptsforpatients:Is one reallya sub-concept
Are theytwo simpleconcepts?Does crampingdescribe
severe abdominalpain? Is crampingin termsof pain
and location,the same abdominal
descriptors,
frequency,
Are
there
other
relatedtoIBS? The
pain?
any
painsensations
researcher
thencomparesand contrasts
whatdifferent
participants
say to seekconsensus(and deviantcases)- often
referred
toas theconstant
method.
Whendeviant
comparison
casesarefound,
theresearcher
seekstounderstand
why.One
moresubjectsorre-analyze
thedatatosearch
mayinterview
forclues thatexplainanydeviationsand theirmagnitudes.
Aftercomparingand contrasting
the data multipletimes
with
the group would then
(ideally
many researchers),
of concepts,associated
develop a conceptualframework
anditemsthatmightmeasurethem.Examples
sub-concepts,
of conceptualframeworks
can be foundin Patricket al.
(2007) andtheFinalGuidanceto Industry
[1,6].
To makemoreclearhow we appliedourapproachto a
researchquestion,we willuse thestudydescribedbelowas
an exampleintherestofthisarticle.In 2008,we conducted
a focusgroupstudyto developa PRO instrument
forirritable bowel syndrome-constipation
(IBS-C)
predominant
and irritable
bowel syndrome-diarrhea
(IBSpredominant
measureof
D). Therewas a needfora newcomprehensive
thesignsandsymptoms
ofIBS inwhichidentified
concepts
would achieve saturation,
conceptswould be clear, the
measurecouldcaptureclinicallymeaningful
changes,anda
meaningfulresponderdefinitioncould be defined.Our
objectivewas todescribetheexperienceoftheseconditions
andtheirtreatments
andtheimpacton a subject'sdailylife.
SamplingstrategiesforPRO development
In a typicalqualitativestudy,one uses theoretical
or purposefulsamplingratherthanprobability-based
representative samples[21]. Representative
experiences,not representativepopulations,are the goal. One may choose to
interview
severalsubgroupswithdifferent
experiencesof
thecondition.Dependingon samplehomogeneity,
itssize
ten to twelvepersonsper group.
may be approximately
Relevantpatientsociodemographic
and/orclinicalcharacteristicsare used to sample and incorporateinclusion/
exclusioncriteria.
Thesemayinclude,forexample,gender;
) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1092

Qual Life Res (2010) 19:1087-1096

and/or
duration
ofsymptoms;
different
frequency,
severity,
treatment
and
treatments
with
different
side
effect
regimens;
Inclusion
and
exclusion
criteria
are
similar
to
profiles.
very
thetargetpopulationthatis to be used in a clinicaltrial.
UsingtheIBS example,one wouldtryto ensurethatthe
and severecases, among
sampleincludedmild,moderate,
otherconsiderations,
so as to understand
theexperienceof
of
Thiswouldallow
different
levels
of
IBS.
having
severity
thedevelopment
of a PRO measurethatcould potentially
andworsening.
capturebothimprovement
Qualitativeinterviewing
uses focusgroupmethodsor inWhethertheresearcher
the developmentof a truly
individual
interviews,
depth
open-endeddiscussionguide is essentialto ensurethat
yielddata thatreflectthe subject's
qualitativeinterviews
or discussionguide
experiencewithoutundueinterviewer
unlikethe
bias. The interviewmustbe semi-structured,
PRO itemsthateventuallyderivefromit. The questions
do not have to be asked verbatimor exactly
themselves
Researchersmightuse
foreveryparticipant.
sequentially
other
instruments,
conceptsfromalready-used
sensitizing
qualitativeliteratureexploringthe same phenomenon,
or interviews
oftheclinicalcondition,
acquiredknowledge
advocatewaitinguntil
withclinicians.Some researchers
datacollectionand analysesare completebeforeconductreviewto avoid bias.
inga literature
focus groups
Four conceptelicitationgender-specific
the sympon
and
focused
wereconducted
understanding
tomsof IBS-C thatsubjectsexperience.Gender-specific
groupswerechosenbecauseofthesensitivenatureofIBS.
conThe interview
guidewas developedusingsensitizing
with
in
discussion
and
on
IBS
in
the
literature
found
cepts
for
much
clinicians.Sensitizingconcepts,a starting
point
a
of qualitative
research,
giveguidancewhenapproaching
what
do
not
but
or
prescribe
phenomenon experience
the researchershould see; rather,they suggestwhere
a researchermightwant to look [40]. The moderator
interview
followeda semi-structured
guide thatincluded
is a good day with
"What
such
as
questions
open-ended
IBS?" followedby "Whatis a bad day withIBS?" We
whatmadea good day or
in reporting
werenotinterested
and dimensionsof
bad dayper se butin whatsymptoms
from
bad days.
differentiated
thosesymptoms
good
Data analysis
Miller and Crabtreelumpedgroundedtheoryand phein an "editing"data analysisstyle
nomenology
together
to
the researcheras a textinterpreter
thatincorporates

determine
units,developcategories,
identify
interpretively
connections
betweensuchcategories,
and verify
theinitial
datafinding
This
iterative
[18,p. 20].
processyieldsa final
and detailsthedata collectionand
reportthatsummarizes
thisapproachby
analysis.Millerand Crabtreesummarize
entersthetextmuch
metaphorically
noting,"theinterpreter
like an editorsearchingformeaningful
segments,
cutting,
untilthereducedsummary
reveals
pasting,andrearranging
theinterpretive
truthin thetext"[18, p. 20]. Whilephenomenologyprovidesa lens with which to view the
experiences of subjects using narrativedescriptions,
groundedtheoryfacilitatesseeks to collect and codify
To
experiencesintoa meaningful,
conceptualframework.
with
PRO
content
developmeaningful
endpoints
validity,
researchers
needto incorporate
bothapproaches.
It is in thedata analysisof qualitativeresearchwhere
for the
groundedtheorymethodsexhibittheirstrength
of conceptualframeworks.
One buildsthese
development
conceptualframeworks
byinduction,
movingfromspecific
to higherlevel conceptsto even moregeneralconcepts
we have reliedon
(domains).In ourresearchexperiences,
The
Straussand Corbin's[37] techniquesand procedures.
researchers
as
collect
data
[37]. Here, the
analyze they
in
an
iterative
researcher
engages
process,withthemodus
operandi coding system,and either manually codes
"chunks"of transcribed
textor "quotations,"or does so
using the increasingpopularsoftwarepackagesthataid
data analysisand theorybuilding.Due to computerized
data analysis,qualitativeresearchhas become morerigwhile
and mostimportantly,
orous,efficient,
transparent,
less
time.
consuming
In theIBS study,following
eachfocusgroup,videotapes
characteristics
verbatim.All identifying
were transcribed
from
wereremovedfromtranscripts.
First,thetranscripts
thefocusgroupswereenteredintoa qualitativesoftware
the
package,ATLAS.ti.ATLAS.tiis designedto facilitate
data
of
and
retrieval
using
qualitative
storage,coding,
Boolean operators[41].
As the coding schemeis developed,some codes are
and someare not.Forexample,
repeatedacrossinterviews
code a researcher
thefirst
mayuse whencodinga transcript
forIBS-D simplymightbe "pain." But
froman interview
upon examinationof quotes that have been coded as
"pain," differentcategoriesthat representa concept
dimensionmay arise, e.g., "pain intensity,""pain fre"As
quency,""pain duration."As Charmazsummarizes,
to
to
a
the
code
raise
categoryyoubegin:(1) explicate
you
its properties;(2) to specifyconditionsunderwhich it
arises, is maintained,and changes; (3) to describeits
consequences;and (4) to showhowthiscategoryrelatesto
othercategories"[21, p. 41].
andinterpretive
Overall,theiterative
processofconstant
comparisonanalysiswas used to develop or supporta

) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

Qual Life Res (20 10) 19: 1087- 1096

1093

forIBS-D and IBS-C. In thisana- goodday/bad

conceptualframework
dayquestions,theywouldcode,forexample,
are
and
confor
IBS-D
lyticprocess,subjectquotations compared
severity:"...if you're going to have another
in
several
earlier
bout
of
trasted
diarrhealateron in theday,oryou'regoingto have
ways: iteratively
by comparing
forexample,severity stabbingpainwhileyou'retrying
and laterinterviews;
to do yourjob, and then
by sub-groups,
of thecondition;and by concepts,e.g., whether"cramp- haveto leave"; or forfrequency:
"Whenyouhavediarrhea
ing"and "abdominalpain" arethesameconcept.Through like threeor fourtimes."The researchermightwritea
such comparisons,it became clear thatthatabdominal memostating,
It seemslikesubject728 is talkingaboutthefeelingto
discomfort,
partof theclinicaldiagnosticcriteriaof both
IBS-C and IBS-D, was a multidimensional
For
to hold it. Lots of inaudiblebut it seems that728,
concept.
trying
meanta mildpain, full- 723, 735, and 722 mightbe talkingaboutthepainwiththe
subjectswithIBS-C, discomfort
as I continueto
ness, and bloating.For subjectswithIBS-D, abdominal cramping.I needto checkthisassumption
discomfort
state(i.e., relatesto collectand analyzedata.
appearsto be an affective
an emotional
rather
thana
Such memosshouldbe used to makecomparisonsiniresponseoffeelingembarrassed)
itself
that
results
from
various
and
then categories,and finally
symptom
symptoms
tiallybetweenrespondents,
associatedsensations(e.g., mild pain, bloating,and the concepts[36].
immediate
needtogo). SubjectswithIBS-D didnotuse the
As theyfurther
compareand contrastdata,researchers
word"discomfort"
to denoteinformation
perse butspokeof theuncomfortable may need codes forsubcategories
above.
suchas thewhat,when,where,why,and howa conceptis
aspectsof IBS-D mentioned
A preliminary
was conducted likelyto occur[37]. Thisprocesswillaid development
ofa
analysison thetranscripts
to identify
theconcepts(i.e., rootconceptsor symptoms conceptualframework
thatincludesspecificPRO concepts
experienced
by subjectswithIBS) relatedto theresearch one wants to focus on in termsof potentialtreatment
A
list
of everywordand its frequency
in theset benefit.This interpretative
of
question.
processis the cornerstone
oftranscripts
was generated.
Each wordwas reviewed,and qualitativeresearch,and it is necessaryto condensethe
whena wordappearedas a potential
conceptbased on the large volumeof textualdata. It is essentialto capturing
team's knowledgeof clinical indicatorsof IBS (e.g., subjects'meaningof feelingsand impactsof a condition
abdominalpain), it was used to populatea list of root and itstreatment.
concepts.Wordwithsame rootsor whichwereconceptuto shortenthe list.
ally equivalentweregroupedtogether
This exercise startedthe coding scheme that reflected Theoreticalsaturation
potentiallyimportant
concepts based on the subjects'
rather
than
based Accordingto Glazerand Strauss,researchers
words,
needto show
beingpredefined
bytheresearcher
on his/her
of
thattheyhavecoveredtheirtopicin depthby havingsufknowledge thecondition,to avoid any bias
ficientcases to exploreand withwhichto elaboratetheir
[42].
wereassignedto different
researchers
fora
Transcripts
categories(or simpleconcepts)fully[36]. This is referred
reviewpreceding
In theIBS study,theachievement
of satthorough
anycodingto give thecontext to as saturation.
ofthesubject'sresponses.Videosofthefocusgroupswere urationwas documented
to showthatall theconceptsthat
observedby different
researchers
to look fornuancesin were importantfor the subjects were consideredfor
and
other
visual
cues.
Visual cues, for inclusionin the conceptualframework
of a PRO instrubody language
included
confirmation
of
the
location
of thebody ment.Saturationis achieved if all conceptsand their
example,
in whichabdominalpain in IBS was experiencedor the relationships
witheach other(how theymaybe grouped)
See Table 2 for
apparentanguish experiencedwith symptomscausing are includedin theconceptualframework.
socialembarrassment
suchas flatulence.
A codingscheme a hypothetical
framework
forIBS-C.
withreliability
was generated
of more
The achievement
of saturation
ensurestheadequacyof
bytheparticipation
thanone coder to processthe firstfew transcripts,
and thesamplesize; if notachieved,new conceptsemergein
and further
subsequent
interviews
groupdiscussionsoftheinterviews,
transcripts, the finalfocusgroupor interview,
and codes. Code agreements
and disagreements
weredis- must be conducted.When concepts and sub-concepts
cusseduntilconsensuswas achieved.
cannotbe further
specifiedwithadditionalanalysisor new
As researchers
workwiththetext,theywritememosin data collection,saturation
is achieved[37, 43]. The quanwhichtheyidentify
ofa category tityof data in a categoryis nottheoretically
(characteristics
to
properties
important
thatdefinesor givesit meaning)and give theirunderlying theprocessof saturation,
and richnessof data is derived
abouthowcategories
assumptions
developorchangeeither fromdetaileddescriptionand not the numberof times
withina respondent's
textor across respondents
or time something
is stated[43]. In qualitativeresearch,"it is often
coded subjectresponsesto the theinfrequent
periods.Whenresearchers
that
gemthatputsotherdataintoperspective
Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1094

Qual Life Res (2010) 19:1087-1096

Table 2 Example of conceptualframework

forIBS-C
Domains
Primary
Constipation
symptoms symptoms

Concepts
- Spontaneousincompletebowel
movement(SICBM)
-> Completespontaneousbowel
movement(SCBM)
-> Unsuccessfulbowel movement
(BM)
- Straining

- Stool consistency
Otherabdominal - Abdominalpain
symptoms
_ Bloating
- Abdominalfullness

becomesthecentralkeyto understanding
thedata and for
the
model"
Table
3 shows an
[43, p. 148].
developing
saturation.
of
one
to
example
way present
is nota frequency
count[36]. To
Note thatsaturation
displaytheresultsof ourevaluationof saturagraphically
tionwe needto showthatit is nota staticbuta dynamic
natureof
concept.This graphicmustdisplaytheiterative
To
do
thedata
data
collection
and
so,
analysis.
qualitative
and
the
wereorganizedin chronological
order,
progression
withineach focusgroupor interof conceptidentification
Thenconceptselicitedacrosssubviewwas documented.
foreach focusgroupusing
jects werecomparedseparately
a stepwiseapproach:conceptselicitedby the firstset of
subjects(focusgroup1) were comparedto the concepts
elicitedby thenextset of subjects(focusgroup2). The
listof conceptselicitedfromthefirsttwo
comprehensive
setsofsubjectswas comparedto conceptselicitedfromthe
thirdsetof subjects(focusgroup3); thisprocesscontinued
withthe fourthfocusgroup(see Table 3). A domainor
if the
simpleclear conceptwas consideredforsaturation
but
not
the
last
focus
in
at
least
one
was
elicited
concept
was
information
of
interviews
and
or
set
enough
group
of
themeaningand importance
elicitedto fullyunderstand
If theconceptwas elicitedonlyin
theconceptto patients.
was considered
the last focusgroup,thenthe saturation
data collectionthough
further
and therefore,
questionable,
The unit
wouldbe recommended.
focusgroupinterviews
of analysisforthe saturation
gridwas each focusgroup
the
(n = 4) forthe IBS study.For individualinterviews
forexample
setsofinterviews,
unitofanalysisis preferably
wereconducted.
3 setsof fiveif 15 interviews
Questionsof reliabilityand validityof results
When qualitativeresearchers
speak of validity,theyare
and
withcredibility,
concernedprimarily
transferability,

Table 3 Hypotheticalconceptsaturationof conceptsin IBS-C focus

groups
Concepts

Focus groups

BM consistency(liquid)
BM consistency(solid)

0 vs. 0 0 vs. 0

0 vs. 1

No

1 vs. 1 2 vs. 0

2 vs. 1

Yes

BM evacuation
(incomplete)
BM evacuation(none)

1 vs. 1 2 vs. 1

3 vs. 1

Yes

1 vs. 1 2 vs. 1

3 vs. 0

Yes

BM frequency
BM size

1 vs. 1 2 vs. 1

3 vs. 1

Yes

1 vs. 1 2 vs. 0

2 vs. 0

Yes

Straining

1 vs. 1 2 vs. 1

3 vs. 1

Yes

Saturation

1 vs. 2 1-2 vs. 3 1-3 vs. 4

NoteBM color and BM odorwereonlymentionedby one subject,and

mucus was mentionedby threesubjects in only one focus group;
these were not consideredfor inclusionin the saturation
therefore,
grid; rectal fullnessand rectal pressurewere relatedto incomplete
bowel movementand straining,
respectively

trustworthiness
to validityas
[44]. Sandelowskireferred
where
a
"stable"
interpretive
validity,
categoryis confirmedby data [45]. Rigoroususe of theproceduresand
techniquesdelineatedherein,in conjunctionwithdocumentationof theiruse, will supportthe validityof the
conceptualframework
developedand the itemsthatare
formedfromit.
in qualitativeresearchis conThe issue of reliability
troversial;however,workingiterati
vely with teams to
schemes
and
elaboratingthe data into
develop coding
andconceptualframeworks
adds
categories,
subcategories,
to thenotionthattheresultsare reliable.In this
credibility
sense,if anothergroupwereto collectand analyzethese
data in an identicalmanner,the outcomewouldbe very
or
similarto that in the initial study(reproducibility
the
and
intraWe
that
one
test
interrepeatability). suggest
raterreliabilityof the coding schemeas a measureof
Ifresourcesand/or
timeprohibit
this,one should
reliability.
or use ranhave morethanone coderprocessa transcript
to discussany
domsamplesoftextfromseveraltranscripts
discrepanciespendingconsensuson a codingscheme.In
reviewed
thecase of theIBS study,twoseniorresearchers
betweenthem.
thecodingand discussedanydiscrepancies
thedecisionKirkand Millersuggestedthatdocumenting
team
as
of
the
research
theyworktoward
makingprocess
its conclusionallows thereaderto evaluatethereliability
of theresults[46]. An exampleof a codingdecisionin the
IBS studyfollows:Patientsused the word "urgency"in
bothIBS-C andIBS-D andwerecodedwiththesamecode.
Furtheranalysissuggested,however,thaturgencyof the
conwas a different
immediateneed to use thebathroom
of
In
the
sense
in
two
disorders.
the
IBS-D,
urgency
cept
or
actuallydid meanthephysicalneedto use thebathroom
was associatedwith fear of accidentallymovingone's

} Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1095

Qua! LifeRes (2010) 19:1087-1096

todevelopPROs
ofqualitative
methods
Table 4 Keyattributes
Method

KeyAttribute

oftheexperience
Sampling Representative
elicitation
of spontaneous
responses
Open-ended
Interviewing
at leasttwocoders;
Constant
comparison;
Analysis
harmonization
Saturation Iteratively
achieved;nota frequency
count;new
framework
conceptdoesnotaddtoconceptual
coders
and
within
a coder's
between
Reliability Agreement
coding
oftheconstruction
oftheconceptual
Documentation
Transparency
of study
framework
fromthebeginning

anddomains.Saturation
in
developconcepts,sub-concepts,
bothstudieswas evaluated,andtheconsistency
ofconcepts
andsubjects'meaningbetweenthedatasetswas confirmed.
Conclusion

This articlesoughtto presentmethodsto develop PROs

through
rigorousqualitativeresearch.This notonlyfillsa
butalso movesbeyondarticles
gap in thePRO literature
thatsuggestcriteriato assess qualitativeresearchin the
healthcarefieldingeneral.Effective
researchis
qualitative
a crucialcomponent
of theobjectivesand requirements
of
PRO development
and validation:to developa conceptual
bowels.In IBS-C on theotherhand,urgencywas in effect framework;to use appropriate,meaningful,subjectto have a BM, a
friendly
wordingwhendevelopingthe items,responses,
feelingafraidof missingtheopportunity
and
recall
and to testtheir
with
IBS-C.
event
for
periodswithinquestionnaires;
patients
positive
and finally,
to ensurethat
of the facevalidityandcomprehension;
See Table 4 fora reviewof the key attributes
no
items
have
been
deleted
on statistics
in
this
to
methods
article
(based
important
qualitative
presented
develop
the
validation
PROs.
alone) during quantitative
phase. Although
researchers
omit this latteruse of qualitative
frequently
researchor have overlookedits applicability,it is the
crucialfinalstep to ensurethatthe instrument
possesses
Triangulation
contentvalidity.Rigorous,well-documented
qualitative
refersto the combination
of data sources, researchprovidesevidencethattheconcepts,domains,and
Triangulation
areappropriate,
different
on a phenom- itemsin a PRO instrument
researchers,
comprehensive,
multipleperspectives
ortheuse of multiple
enonof interest,
methodsto arriveat and interpretable
[49].
conclusionsabouta researchquestion[47, 48]. In qualiunderthe termsof the
tativeresearch,
and Open Access This articleis distributed
triangulation
givesgreaterperspective
CreativeCommonsAttribution
Noncommercial
Licensewhichperallows formorecredibility
in one's findings.
When the mits noncommercial
and reproduction
in any
use,distribution,
any
frommethodsand datasourcesconverge,one has
theoriginal
andsourcearecredited.
findings
medium,
author(s)
provided
moreconfidence
in them;whentheydiverge,thispresents
an opportunity
to takea closerlook at all data to gain a
betterunderstanding
of the phenomenonbeing studied References
[47].
1. Patrick,
D., Rock,E., O'Neill,R., Powers,J.,Scott,J.,Kennedy,
Findingsfromfocusgroupdata on IBS were trianguoutcomesto support
medical
D., et al. (2007). Patientreported
lated (analyzediteratively)
withfindingsfromcognitive
claims:FDA perspective.
Valuein Health,10,
product
labelling
125-137.
interviews
in IBS-C and anothersetof focusgroupdata in
2.
E. (2005). DevelopingQuestionnaires.
In P. Fayers&
McColl,
IBS-D. Cognitiveinterviews
consistof usingverbalprobR. Hays(Eds.),Assessing
oflifeinclinicaltrials(2nded.,
quality
aboutitems
ingtechniquesto elicitrespondents'
thinking
Press.
pp.9-23). OxfordUK: OxfordUniversity
in a questionnaire
to identify
and
3. Sireci,S. G. (1998). The construct
of contentvalidity.
Social
problems
supportthe
Indicators
Research,
45, 83-117.
contentvalidityof questions[11]. In theIBS-C cognitive
4. Ware,J. E, Jr,Keller,S. D., Hatoum,H. T., & Kong,S. X.
a set of IBS
interviews,
respondents'
thinking
regarding
healthindex(ASHI): I.
(1999). The SF-36 arthritis-specific
itemswas elicited,including
theirrelevance,interpretation,
andcross-validation
ofscoring
Medical
Development
algorithms.
and importance.
Care,37(5 Suppl),MS4O-MS50.
5. Meynck,
J.(2007).Whatis goodqualitative
research?
A first
The inclusion/exclusion
thedemographic
charcriteria,
step
towards
a
comprehensive
approachtojudgingrigorous
quality.
acteristics
ofthedifferent
studysamplesand themeanIBS
Journal
11(5),799-808.
ofHealthPsychology,
level in each set of focusgroupswerecompared. 6. US
severity
ofHealthandHumanServices.(2009).Guidance
Department
similarin thedifferent
forindustry
outcomemeasures:
Use in Medical
Findingtheparticipants
relatively
patient-reported
Product
to
datasets,we continued
withthetriangulation
Development
Suppor
LabellingClaims.
Each
process.
Issuesand Patient-Reported
Outdata set was approachedin the same way as described 7. Revicki,D. A., & Regulatory
comesTaskForcefortheInternational
SocietyforQualityofLife
herein.A codingschemewas developedand harmonized.
Research.(2007). FDA draftguidanceand health-outcomes
Coded quotationswere compared and contrastedto
research.
Lancet,569(9561),540-542.
) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions

1096

Qual Life Res (2010) 19:1087-1096

of Healthand HumanServices.(2006). Guidance

8. US Department
forIndustry:Patient-Reported
Outcome Measures: use in Medical ProductDevelopmentto SupportLabeling Claims. Internet.
http://www.fda.gov/cder/guidamce/5460dft.pdf.
9. Presser,S., Rothgeb,J.M., Martin,J.,Singer,E., Lessler,J.T., &
Martin,E. (2004). Methods for testingand evaluating survey
questionnaires.Public OpinionQuarterly,68, 109-130.
10. Tourangeau,R. (2000). Impact of cognitivemodels on survey
In R. Tourangeau,L. J.Rips, & K. Rasinski(Eds.),
measurement.
The psychologyof surveyresponse (pp. 3 13-343). Cambridge,
UK: CambridgeUniversityPress.
in practice.In Cogni11. Willis, G. (2005). Cognitiveinterviewing
tive. Interviewing(Ed.), A tool for improvingquestionnaire
design(pp. 42-62). ThousandOaks, CA: Sage Publishers.
12. Kitzinger,J. (1995). Qualitative research. Introducingfocus
groups.BMJ,311(1000' 299-302.
13. Pope, C, Ziebland,S., & Mays,N. (2000). Qualitativeresearchin
healthcare. Analysingqualitativedata. BMJ, 320(7227), 114116.
14. Hodges, B. D., Kuper, A., & Reeves, S. (2008). Discourse
analysis.BMJ,337, a879.
15. Tong, A., Sainsbury,P., & Craig,J.(2007). Consolidatedcriteria
forreporting
qualitativeresearch(COREQ): A 32-itemchecklist
Journalof Quality
forinterviewsand focusgroups.International
in Health Care, 79(6), 349-357.
16. Acquadro,C, Berzon,R., Dubois, D., Leidy,N. K., Marquis,P.,
thepatient'sperspective
Revicki,D., et al. (2003). Incorporating
An ad hoc taskforce
intodrugdevelopmentand communication:
outcomes (PRO) Harmonization
reportof the patient-reported
February16,
groupmeetingat thefood and drugadministration,
2001. Value in Health,6(5), 522-531.
17. Leidy, N. K. (2006). Evolving concepts in the measurementof
effects.Proceedingsof theAmericanThoracicSociety,
treatment
3(3), 212-217.
18. Crabtree,B. F., & Miller, W. L. (1992). Doing qualitative
research: Research methodsfor primarycare. London: Sage
Publications.
19. Sandelowski,M. (2004). Using qualitativeresearch.Qualitative
Health Research,74(10), 1366-1386.
20. Banister,P., Burman,E., Parker,I., Taylor, M., & Tindall, C.
(2006). Qualiative methodsin psychology:A research guide.
Berkshire,UK: Open UniversityPress.
21. Charmaz,K. (1995). Groundedtheory.In J. A. Smith,R. Harre,
methodsin psychology
& L. Van Langenhove(Eds.), Rethinking
(pp. 27-49). London: Sage.
22. Shaw, R. (2001). Why use interpretative
phenomenological
analysisin healthpsychology?Health PsychologyUpdate,70(4),
48-52.
23. Smith,J.A., Harre,R., & Van Langenhove,L. (2005). Rethinking
methodsin psychology.London: SAGE Publications.
An overviewof
24. Collins,D. (2003). Pretesting
surveyinstruments:
cognitivemethods.Qualityof LifeResearch,72(3), 229-238.
25. Podsakoff,P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff,N.
P. (2003). Common methodbiases in behavioral research: A
critical review of the literatureand recommendedremedies.
JournalofAppliedPsychology,88(5), 879-903.
26. Schaeffer,N., & Presser, S. (2003). The science of asking
questions.AnnualReviewof Sociology,29, 65-88.
of survey
27. Tourangeau,R. (2000). Respondents'understanding
questions.In R. Tourangeau,L. J.Rips,& K. Rasinski(Eds.), The
psychologyof survey response (pp. 23-61). Cambridge,UK:
CambridgeUniversityPress.

28. Tourangeau,R. (2000). Factual judgmentsand numericalestimates. In R. Tourangeau,L. J. Rips, & K. Rasinski(Eds.), The
psychologyof surveyresponse (pp. 135-164). Cambridge,UK:
CambridgeUniversityPress.
29. Tourangeau,R. (1984). Cognitivescienceand surveymethods.In
T. Jabine,M. Straf,J.Tanur,& R. Tourangeau(Eds.), Cognitive
aspects of surveydesign: Buildinga bridgebetweendisciplines
(pp. 73-100). Washington,D.C.: NationalAcademyPress.
30. Armes,J.,Krishnasamy,M., & Higginson,I. (2004). Fatigue in
cancer. Oxford,UK: OxfordUniversityPress.
31. Dixon-Woods, M., Shaw, R. L., Agarwal, S., & Smith,J. A.
(2004). The problemof appraisingqualitativeresearch.Quality&
Safetyin Health Care, 75(3), 223-225.
32. Wu, H. S., & McSweeney, M. (2004). Assessingfatiguein persons withcancer: An instrument
developmentand testingstudy.
Cancer, 101(1), 1685-1695.
33. Liamputtong,P., & Ezzy, D. (2005). Qualitativeresearchmethods (2nd ed. ed.). Melbourne,Australia:OxfordUniversity
Press.
34. Schwartz,C, Sprangers,M. A., & Fayers,P. (2005). Response
shift:You knowit's there,buthow do you captureit?Challenges
for the next phase of research.In P. Fayers & R. Hays (Eds.),
Assessingqualityof lifein clinical trials(2nd ed., pp. 275-290).
Oxford:OxfordUniversityPress.
35. Girot,E. A. (1993). Assessmentof competencein clinical practice: A phenomenologicalapproach.Journalof AdvancedNursing, 18(1), 114-119.
36. Glaser, B. G., & Strauss,A. (1967). The discoveryof grounded
theory:Strategiesforqualitativeresearch.Chicago: Aldine.
37. Strauss,A., & Corbin,J. (1998). Basics of qualitativeresearch:
Techniquesand proceduresfor developinggroundedtheory(2nd
ed. ed.). London: Sage.
38. Sandelowski. (7-16-2000). 5th Annual Summer Institutein
Qualitative Research, Universityof NorthCarolina School of
Nursing,Chapel Hill, NC.
39. Charmaz,K. (1991). Good days, bad days: The self in chronic
illnessand time.New Brunswick,NJ: RutgersUniversityPress.
40. Bowen, G. A. (2006). Groundedtheoryand sensitizingconcepts.
InternationalJournalof QualitativeMethods,5(3), 12-23.
41. Weitzman,E. A., & Miles, M. B. (1995). Computerprogramsfor
qualitativedata analysis. London: Sage Publications.
42. Miles, M. B., & Huberman,A. M. (1994). Qualitative data
analysis. London: Sage Publications.
43. Morse, J. M. (1995). The significanceof saturation.Qualitative
Health Research,5(2), 147-149.
44. Golafshani,N. (2003). Understanding
reliabilityand validityin
qualitativeresearch.The QualitativeReport,8(4), 597-607.
45. Sandelowski,M. (2000). Combiningqualitativeand quantitative
sampling,data collection, and analysis techniques in mixedmethodstudies.Researchin Nursingand Health,23(3), 246-255.
46. Kirk, J., & Miller, M. L. (1986). Reliabilityand validityin
qualitativeresearch.London: Sage.
47. Patton,M. Q. (2002). Qualitativeresearch& evaluationmethods
(3rd ed.). London: Sage Publications,Inc.
48. Denzin, N. K., & Lincoln,Y. S. (1994). Handbookof qualitative
research.London: Sage Publications,Inc.
49. Trentacosti,A. M. (2007). Epoetin Alpha: FDA Overview of
Patient Reported Outcome (PRO) Claims, www.fda.gov/
15s 1-09-FDA-Trentacosti.ppt.
ohrms/dockets/ac/07/slides/2007-43
15s 1-09-FDAwww.fda.gov/ohrms/dockets/ac/07/slides/2007-43
Trentacosti.ppt.

) Springer

This content downloaded from 168.176.5.118 on Mon, 27 Apr 2015 18:44:28 UTC
All use subject to JSTOR Terms and Conditions