Critical Appraisal

Dr A C J Hutchesson
Chair of Examiners’ Panel, FRCPath
(Clinical Biochemistry)

Critical Appraisal - what is it?

• The process of carefully and systematically

examining research to judge:
– Trustworthiness
– Value and relevance in a particular context
• Use of explicit, transparent methods
• A central part of systematic review
“If healthcare professionals and patients are
going to make the best decisions they need to
be able to:
• Decide whether studies have been undertaken
in a way that makes their findings reliable
• Make sense of the results
• Know what these results mean in the context of
the decision they are making.”

Hill A, Spittlehouse C. What is clinical appraisal? Hayward Medical Communications, 2009

What evidence have you got for that?

It’s in the newspapers.

So’s my horoscope.
Consider …

• Validity
– Has the research been conducted in such a
way as to minimise bias?
• Results
– If so, what do the results show?
• Relevance
– What do the results mean for the particular
patient or context in which a decision is being
made?

Validity - research question

• Clearly stated?
• Importance? (e.g. majority or minority of
patients?)
• Change in standard treatment since
study commencement?
 Validity - Research question

• Karl Popper
– Hypothesis testing

• Experimentum crucis
– E.g: homocysteine and heart disease

• “Black swans”

Validity - Study design

• Different questions require different designs!

– Randomised controlled trial treatment effectiveness
– Cohort study prognosis
– Cross-sectional prevalence
– Qualitative patient experience
Validity - Bias
• Selection bias
– Allocation to comparison groups
• Performance bias
– Unequal provision of routine care
• Detection bias
– Assessment of outcome
• Attrition bias
– Occurrence and handling of deviations from protocol
and loss to follow-up
• Conflict of interest

Validity - Subjects

• Inclusion/exclusion criteria?
– Uniform group/avoid confounding factors
– Clinical validity/recruitment rate

• Number (statistical power)

– Type I/ error (accepting -ve result)
– Type II/ error (rejecting +ve result)
Results - Measures

• Accurate, precise?
– E.g: BP measurement
• Confounding factors?
• End-point v. surrogate?
– E.g: carotid IMT v. CHD mortality
HbA1c v. complication rates
• Relevant?

Results - Analysis

• Intention-to-treat
• Appropriate statistics?
– Parametric v. non-parametric
– Univariate v. multivariate (?model)
– Multiple comparisons
• p-value v. effect size (CI, OR, NNT, etc)
Results - Presentation
• Numbers screened, randomised,
completed
• Results:
– Descriptive statistics
– Figures, diagrams, charts
• Absolute v. relative change
– E.g: 25% increase: - 20% 25%?
0.04% 0.05%?

Relevance

• Population studied?
• Implications of results?
• Balanced interpretation
• Other studies - agreement and differences
– What is already known?
– What does this study add?
Checklists

• Appropriate for study design.

• E.g: CASP, SIGN guideline checklists:
• http://www.sign.ac.uk/guidelines/fulltext/5
0/checklist2.html

Systematic review

• Exhaustive summary of relevant literature

• Objective, transparent process
• Eligibility criteria defined
• Levels of evidence (RCT expert opinion)
• Publication bias (“file drawer problem”)
 Meta-analysis
• Statistical method to combine results from several
studies
– Increased power
– Generalisation across population
– Control for between-study variation
– Control for local factors
– Results variability v. sample diversity?
– Derivation/testing of factors/effect size parameters
– Moderators to explain variation
– Publication bias

Meta-analysis

• Good meta-analysis of bad studies

= bad result (GIGO)
• Meta-analysis of small studies
single large study
• Presentation:
– Forest plots - effect size
– Funnel plots - publication bias
Critical Appraisal and FRCPath
• Introduced Spring 2011 (for Clinical Biochemistry)
• Part 2, Module 2
– Cases and Critical Evaluation paper
• A. Six cases (50% marks)
• B. Two (real) journal articles (50% marks)
– one analytical; e.g. Annals, Clin Chem
– one clinical; e.g. Lancet, NEJM, specialist
• Answer all questions (no choices)
• 3 hours; suggest 1 hour for cases

Critical Appraisal and FRCPath

- Preparation
• College website: example questions
http://www.rcpath.org/examinations/past-and-sample-
papers/past-and-sample-papers.htm
• Reading
– Journals; Annals, Clin Chem, BMJ, Lancet …
– Books; e.g. Greenhalgh, T. How to read a paper
• Journal clubs
• Intray exercises:
– Guideline assessment
– Guideline preparation
– Peer review