Systematic reviews and

Meta-analyses
Dr Paul Watts
 Experimental Studies e.g. Randomised
controlled trial
Present Future
Intervention Outcome / Disease
e.g. Medicine / Social e.g. Depression
Prescription Has
outcome
Receives

Compare
Population intervention Doesn’t
have
with the outcome
disease /
outcome Has
outcome
No
intervention
Doesn’t
(control) have
outcome
 Study A: A hypothetical intervention
First, calculate proportions:
Disease A/(A+B) = 62.5%
C/(C+D) = 87.5%
Yes No
A B (multiply by 100 for
percentages)
Intervention 125 75
Next, calculate the odds ratio:

C D Odds Ratio = (A / C) / (B / D)

Control 175 25 Which is the same as:

Odds Ratio = (A*D) / (B*C)

76% reduction in odds of disease OR = 0.24

compared to control group
 Study B: A hypothetical intervention
First, calculate proportions:
Disease A/(A+B) = 55.4%
C/(C+D) = 57.5%
Yes No
A B (multiply by 100 for
percentages)
Intervention 170 137
Next, calculate the odds ratio:

C D Odds Ratio = (A / C) / (B / D)

Control 157 116 Which is the same as:

Odds Ratio = (A*D) / (B*C)

8% reduction in odds of disease OR = 0.92

compared to control group
 Confidence intervals
• Odds ratios alone are not enough to make a
decision about the effectiveness of an
intervention.
• Is an 8% reduction in the odds of disease a ‘real’
difference?
• Or is this difference just by chance?

• https://www.medcalc.org/calc/odds_ratio.php
 Study A
Question: Is intervention A effective in reducing disease?

Odds ratio Lower Upper Confidence

Confidence Interval (UCI)
Interval (LCI)

Control 1 (reference) - -

Intervention 0.24 0.14 0.40

76% reduction in odds of disease In 95% of samples, the odds ratio

compared to control group would be between the UCI and LCI
 Study A: 95% Confidence intervals
Odds Ratio (estimate)

Lower CI Upper CI

0 0.2 0.4 0.6 0.8 1 1.5 2 2.5 3

Odds Ratio
(Favours intervention) (Favours control)
 Study B
Question: Is intervention B effective in reducing disease?

Odds ratio Lower Upper Confidence

Confidence Interval (UCI)
Interval (LCI)

Control 1 (reference) - -

Intervention 0.92 0.66 1.27

8% reduction in odds of disease In 95% of samples, the odds ratio

compared to control group would be between the UCI and LCI
 Study B
Odds Ratio (estimate)

Lower CI Upper CI

0 0.2 0.4 0.6 0.8 1 1.5 2 2.5 3

Odds Ratio
(Favours intervention) (Favours control)
 Sample size, precision and power
• What would happen if we did the same study,
using a larger sample size?
• Ten times more participants?
• One hundred times more participants?
• Larger sample sizes means increased precision.
 Study B: Effect of increased sample size

100 x sample size (58,000)

10 x sample size (5,800)

Original sample size (580)

0 0.2 0.4 0.6 0.8 1 1.5 2 2.5 3

Odds Ratio
(Favours intervention) (Favours control)
Systematic reviews, evidence synthesis,
meta analyses
• One study alone is not sufficient to draw
conclusions about relationships between
variables or effectiveness of interventions.
• Especially studies of low quality.
• Remember in studies using 95% confidence
intervals (or p-values of 0.05):
1 in 20 perfect studies will be wrong by chance!
• So, we need to combine the results of studies.
The beginnings of research synthesis
• Karl Pearson is probably the first
medical researcher to use formal
techniques to combine data from
different studies (1904):
– He synthesized data from several
studies on efficacy of typhoid
vaccination
• His rationale for pooling data:
– “Many of the groups… are far too small
to allow of any definite opinion being
formed at all, having regard to the size
of the probable error involved.”

Egger et al. Systematic reviews in health care. London: BMJ Publications, 2001.
Prof Archibald Cochrane, CBE
(1909 - 1988)

• The Cochrane Collaboration is

named in honor of Archie
Cochrane, a British researcher.
• In 1979 he wrote, "It is surely a
great criticism of our profession
that we have not organized a
critical summary, by specialty or
subspecialty, adapted
periodically, of all relevant
randomized controlled trials.”

Source: http://www.cochrane.org/cochrane/archieco.htm
 Who does systematic reviews?
• Academics
• Researchers
• The Campbell Collaboration
• The Centre for Reviews and Dissemination
(CRD Guidance)
• The Cochrane Collaboration
• http://www.cochrane.org/multimedia/video/
what-cochrane-collaboration
Steps involved for a Systematic Review
1. Develop an answerable question

2. Check to see if there is a recent systematic review on your question

3. Agree on specific inclusion and exclusion criteria

4. Develop a system to track and record search methodologies, databases searched,

articles reviewed, why articles were included/excluded in final data synthesis

5. Devise reproducible search methods

6. Launch and track exhaustive search

7. Organize search results

8. Reproduce search results

9. Abstract data into a standardized format

10. Synthesize data using statistical methods (meta-analysis)

11. Write about what you found

 Are the studies good quality?
• Critical appraisal tools
http://www.casp-uk.net/#!casp-tools-
checklists/c18f8
Risk of bias assessment tools:
https://www.bmj.com/content/bmj/343/bmj.d5
928.full.pdf
 Meta analyses: Definition
• Meta-analysis: a type of systematic review that
uses statistical techniques to quantitatively
combine and summarize results of previous
research

• A review of literature is a meta-analytic review

only if it includes quantitative estimation of the
magnitude of the effect and its uncertainty
(confidence limits).
 Why is Meta-Analysis Important?
• Researchers used to think the aim of a single study was to
decide if a given effect was "real" (statistically significant).
• When many studies were done, someone would write a
narrative (= qualitative) review trying to explain why the
effect was/wasn't real in the studies.
• In modern systematic reviews, the aim of meta-analyses is
to get the overall magnitude of an effect with precision.
• Each study produces a different estimate of the
magnitude.
• Meta-analysis combines effects from all studies to give an
overall mean effect size.
 Purpose of Meta-Analyses
• Identify heterogeneity in effects among multiple
studies and, where appropriate, provide
summary measure.
• Increase statistical power and precision to detect
an effect.
• Reduce the subjectivity of study comparisons by
using systematic and explicit comparison
procedure
• Identify data gap in the knowledge base and
suggest direction for future research
 Meta analyses: Basic concepts
• The main outcome is the overall magnitude of
the effect.
• It's not a simple average of the magnitude in
all the studies.
• Meta-analysis gives more weight to studies
with more precise estimates.
 Meta analyses: Statistics
• You can combine effects from different studies
only when they are expressed in the same units.
• Meta-analysis uses the magnitude of the effect
and its precision from each study to produce a
weighted mean.
• The weights are often the inverse of the
variance (the square of the standard error) of
the treatment effect, which relates closely to
sample size.
• The usual graph for displaying the results of a
meta-analysis is called a ‘‘forest plot’’.
 Meta analysis of nine studies: Effect of probiotics on
the risk of antibiotic associated diarrhoea
Study No. Odds ratio (95% Cl) Weight (%)

1 0.37 (0.16 to 0.88) 15.1

2 0.46 (0.18 to 1.18) 12.1
3 1.67 (0.47 to 5.89) 3.5
4 0.22 (0.10 to 0.48) 29.9
5 0.88 (0.22 to 3.52) 3.9
6 0.23 (0.09 to 0.56) 21.2
7 0.58 (0.07 to 4.56) 2.2
8 0.25 (0.05 to 1.43) 5.2
9 0.34 (0.09 to 1.38) 7.0

0.37 (0.26 to 0.52) 100%

D’Souza AL, Rajkumar C, Cooke
0 0.2 0.4 0.6 0.8 1 2.5 4 6.5 J, et al. Probiotics in prevention
of antibiotic associated diarrhoea:
(Favours intervention) Odds Ratio (Favours control) meta-analysis. BMJ
2002;324:1361
Other commonly encountered comparative effect
measures

Type of Data Corresponding Effect Measure

• Mean difference (e.g., mmol, mmHg)

Continuous
• Correlation

Dichotomous /
Odds ratio, risk ratio, risk difference
Binary / Categorical

Time to event Hazard ratio

 Relative risk
The impact of fish oil consumption on cardiovascular diseases
The impact of fish oil consumption on Cardio-vascular diseases
 Combining Effect Estimates –
Continuous outcomes
What is the average (overall) treatment-control
difference in blood pressure?

Mean difference 95% Confidence

Study N (mm Hg) Interval

A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

 Simple Average
What is the average (overall) treatment-control
difference in blood pressure?
(-6.2) + (-7.7) + (-0.1)
= -4.7 mm Hg
3
Mean
difference
Study N mmHg 95% CI
A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

 Weighted Average
What is the average (overall) treatment-control
difference in blood pressure?
(554 x -6.2) + (304 x -7.7) + (39 x -0.1) = -6.4 mm Hg
554 + 304 + 39

Mean
difference
Study N mmHg 95% CI
A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

 Forest plots: mean difference

Mean difference

Kelley, G. A., Kelley, K. S., & Jones, D. L. (2011). Efficacy and effectiveness of exercise on tender points in adults with
fibromyalgia: a meta-analysis of randomized controlled trials. Arthritis, 2011.
 Bias in Meta-analyses
• Poor Quality of Trials
– To avoid them, learn more at CONSORT statement
[http://www.consort-statement.org]
• Publication Bias
– study showing beneficial effects of new treatment
more likely to be published than one showing no
effect
– negative trials assumed to contribute less; never
show up in the literature base
– use several approaches to avoid this
– Use Funnel Plots to examine the influence of
publication bias
 Critical Appraisal of Systematic
Reviews
PRISMA:
Preferred Reporting Items for Systematic Reviews
and Meta-Analyses (PRISMA)

http://www.prisma-
statement.org/documents/PRISMA%202009%20c
hecklist.pdf
 Example: Risk Ratios (1)

High physical activity & Cognitive decline (Sofi et al, J Internal Med, 2010;269:107-117)

32
 Example: Risk Ratios (2)

Forest plot of smoking and multiple sclerosis risk. Risk Ratio

Handel AE, Williamson AJ, Disanto G, Dobson R, Giovannoni G, et al. 2011 Smoking and
Multiple Sclerosis: An Updated Meta-Analysis. PLoS ONE 6(1): e16149
 Example: Hazard Ratios
Childhood IQ and risk of mortality

Childhood IQ and risk of mortality (Calvin et al., 2010) 34

Example: standardised mean difference
Relationship between folic acid intake and cognitive function

35
Wald et al, Am J Medicine, 2010;123(6):522-7