Critical appraisal of quantitative research

November 2010 Sarah Lawson Research & Learning Support sarah.lawson@kcl.ac.uk

1

Learningobjectives
Understandtheprinciplesofcriticalappraisalanditsrole inevidencebasedpractice Understandthedifferentlevelsofquantitativeevidence Beabletoappraisequantitativeresearchandjudgeits validity Beabletoassesstherelevanceofquantitativeresearch totheirownwork Knowaboutresourcesavailabletohelpthemtocritically appraiseresearch Beabletocriticallyappraisequantitativeresearchasa group

Whatisevidencebasedpractice?
Evidencebasedpracticeistheintegrationof individualclinicalexpertise withthe bestavailableexternalclinicalevidence from systematicresearch and patientsvaluesandexpectations

Theevidencebasedpractice(EBP) process.
Decisionorquestionarisingfroma patientscare. Formulateafocusedquestion. Searchforthebestevidence. Appraisetheevidence. Applytheevidence.

EBPinpractice
dependinguponspeciality,between50and80per centofall'medicalactivity'isevidencebased. www.shef.ac.uk/scharr/ir.percent.html

Whydoesevidencefromresearchfail togetintopractice?
75%cannotunderstandthestatistics 70%cannotcriticallyappraisearesearchpaper
UsingresearchforPractice:aUKexperienceofthebarriersscale. Dunn,V.etal.

Whatiscriticalappraisal?
Weighingupevidencetoseehowusefulitis indecisionmaking Balancedassessmentofbenefitsand strengthsofresearchagainstitsflawsand weaknesses Assessresearchprocessandresults Skillthatneedstobepracticedbyallhealth professionalsaspartoftheirwork

WhatcriticalappraisalisNOT
Negativedismissalofanypieceofresearch Assessmentofresultsalone Basedentirelyonstatisticalanalysis Onlytobeundertakenbyresearchers/ statisticians

Whydoweneedtocritically appraise?
Itusuallycomesasasurprisetostudentsto learnthatsome(thepuristswouldsay99%of) publishedarticlesbelonginthebinandshould notbeusedtoinformpractice Greenhalgh 2001

Whydoweneedtocritically appraise?
studieswhichdon'treporttheirmethodsfully overstatethebenefitsoftreatmentsbyaround25% Khanetal.ArchInternmed,1996; Maheretal,Lancet1998. studiesfundedbyapharmaceuticalcompanywere foundto be4timesaslikelytogiveresultsthat werefavourabletothecompanythanindependent studies Lexchin etal,BMJ,2003

Sourcesofbias
poorcontrolgroup/controldosage surrogateoutcomes ignoredropouts modifytriallength misusebaselinestatistics statisticsoverload

HowdoIappraise?
Mostlycommonsense. Youdonthavetobeastatisticalexpert! Checklistshelpyoufocusonthemost importantaspectsofthearticle. Differentchecklistsfordifferenttypesof research. Willhelpyoudecideifresearchisvalidand relevant.

Researchmethods
Quantitative Usesnumbersto describeandanalyse Usefulforfinding preciseanswersto definedquestions Qualitative Useswordstodescribe andanalyse Usefulforfinding detailedinformation aboutpeoples perceptionsand attitudes

Levelsofquantitativeevidence.
Systematicreviews. Randomizedcontrolledtrials. Prospectivestudies(cohortstudies). Retrospectivestudies(casecontrol). Caseseriesandreports Opinionsofrespectedauthorities.

SystematicReviews.
Thoroughsearchofliteraturecarried out. AllRCTs (orotherstudies)onasimilar subjectsynthesisedandsummarised. Metaanalysistocombinestatistical findingsofsimilarstudies.

RandomisedControlledTrials (RCTs)
Normaltreatment/placeboversusnew treatment. Participantsarerandomised. Ifpossibleshouldbedoubleblinded. Intentiontotreatanalysis

Cohort studies
prospective groups (cohorts) exposure to a risk factor followed over a period of time compare rates of development of an outcome of interest Confounding factors and bias

Case control studies

Retrospective Subjects confirmed with a disease (cases) are compared with non-diseased subjects (controls) in relation to possible past exposure to a risk factor. Confounding factors and bias

Appraising original research

Aretheresultsvalid? Istheresearchquestionfocused? Wasthemethodappropriate? Howwasitconducted? Whataretheresults? Howwasdatacollectedandanalysed? Aretheysignificant? Willtheresultshelpmyworkwithpatients?

AppraisingRCTs
Recruitment and sample size Randomisation method and controls Confounding factors Blinding Follow-up Intention to treat analysis Censoring

Appraisingsystematicreviews.
Wasathoroughliteraturesearchcarriedout? Publicationbias paperswithmoreinteresting resultsaremorelikelytobe:
Submittedforpublication Acceptedforpublication Publishedinamajorjournal PublishedintheEnglishlanguage

Publicationbias
AllSSRItrialsregisteredwithFDA 37studieswereassessedbyFDAaspositive 36ofthesewerepublished. 22studieswithnegativeorinconclusive resultswerenotpublishedand11were writtenupaspositive. Turneretal.NEJM,2008.

Reviewsingeneralmedicaljournals
50reviewsin4majorjournals19856 Nostatementofmethods Summaryinappropriate Currentsystematicreviewsdonotroutinely usescientificmethodstoidentify,assessand synthesiseinformation (Mulrow,1987)

Istheresearchquestionfocused?

Patient (e.g.child) Intervention (e.g.MMRvaccine) Comparison (e.g.singlevaccines) Outcome (e.g.autism)

Areresultssignificant?
Howwasdatacollected? Whichstatisticalanalyseswereused? Howprecisearetheresults? Howaretheresultspresented?

Intentiontotreatanalyses
Analysingpeople,attheendofthetrial,inthe groupstowhichtheywererandomised,even iftheydidnotreceivetheintended intervention

Statisticalanalyses
Oddsratios,absoluteandrelative risks/benefits,hazardratios/relativehazards Thelikelihoodofsomethinghappeningvs the likelihoodofsomethingnothappening Numberneededtotreat(NNT) Thenumberofpeopleyouwouldneedto treattoseeoneadditionaloccurrenceofa specificbeneficialoutcome

OddsRatioDiagrams.(Blobbograms or
ForestPlots.)

OddsRatioDiagrams
Lineofnoeffect nodifferencebetween treatmentandcontrolgroup Result(blob)totheLeftofthelineofnoeffect =Lessoftheoutcomeinthetreatmentgroup. ResulttotheRightoftheline=Moreofthe outcome. BUT Istheoutcomegoodorbad?

Cardiacdeaths less=good

Smokingcessation more=good

ConfidenceIntervals.
Longerconfidenceinterval=less confidentofresults widerrange. Shorterconfidenceinterval=more confident narrowerrange. Crosseslineofnoeffect/nosignificance= Inconclusiveresults.

Confidenceintervals

PValues.
Pstandsforprobability howlikelyisthe resulttohaveoccurredbychance? Pvalueoflessthan0.05meanslikelihoodof resultsbeingduetochanceislessthan1in20 =statisticallysignificant. Pvaluesandconfidenceintervalsshouldbe consistent

NumberNeededtoTreat
Thenumberofpeopleyouwouldneedto treattoseeoneadditionaloccurrenceofa specificbeneficialoutcome. Thenumberofpatientsthatneedtobe treatedtopreventonebadoutcome.

Areresultsrelevant?
CanIapplytheseresultstomyownpractice? Ismylocalsettingsignificantlydifferent? Arethesefindingsapplicabletomypatients? Arefindingsspecific/detailedenoughtobe applied? Werealloutcomesconsidered?

Thegoodnews!
Someresourceshavealreadybeen criticallyappraisedforyou. Anincreasingnumberofguidelinesand summariesofappraisedevidenceare availableontheinternet.

Summary.
Searchforresourcesthathavealreadybeen appraisedfirst,e.g.Guidelines,Cochranesystematic reviews. Searchdownthroughlevelsofevidence,e.g. systematicreviews,RCTs. Usecheckliststoappraiseresearch. Howcantheseresultsbeputintopractice?

Questions?

39