Journal of Clinical Epidemiology 54 (2001) 10461055

Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis

Jonathan A. C. Sterne*, Matthias Egger
MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, Canynge Hall, Whiteladies Road, Bristol BS8 2PR UK Received 03 August 2000; received in revised form 11 April 2001; accepted 12 April 2001

Abstract
Asymmetry in funnel plots may indicate publication bias in meta-analysis, but the shape of the plot in the absence of bias depends on the choice of axes. We evaluated standard error, precision (inverse of standard error), variance, inverse of variance, sample size and log sample size (vertical axis) and log odds ratio, log risk ratio and risk difference (horizontal axis). Standard error is likely to be the best choice for the vertical axis: the expected shape in the absence of bias corresponds to a symmetrical funnel, straight lines to indicate 95% confidence intervals can be included and the plot emphasises smaller studies which are more prone to bias. Precision or inverse of variance is useful when comparing meta-analyses of small trials with subsequent large trials. The use of sample size or log sample size is problematic because the expected shape of the plot in the absence of bias is unpredictable. We found similar evidence for asymmetry and between trial variation in a sample of 78 published meta-analyses whether odds ratios or risk ratios were used on the horizontal axis. Different conclusions were reached for risk differences and this was related to increased between-trial variation. We conclude that funnel plots of meta-analyses should generally use standard error as the measure of study size and ratio measures of treatment effect. 2001 Elsevier Science Inc. All rights reserved.
Keywords: Funnel plot; Meta-analysis; Randomized controlled trials; Selection bias; Publication bias; Statistical method; Systematic reviews

1. Introduction Funnel plotsscatter plots in which the treatment effects estimated from individual studies on the horizontal axis are plotted against a measure of study precision on the vertical axishave been proposed as a means of detecting publication bias in meta-analysis [1]. In the absence of bias the graph resembles a symmetrical inverted funnel because the treatment effect estimates from smaller studies scatter more widely at the bottom of the graph, with the spread narrowing with increasing precision among larger studies. If there is publication bias because smaller studies which show no statistically significant effects remain unpublished [2,3], then the funnel plot will appear asymmetrical [4,5]. Funnel plot asymmetry cannot, however, be interpreted as proof of publication bias in meta-analysis [6]. Asymmetry could also result from the overestimation of treatment effects in smaller studies of inadequate methodological quality [7]. Furthermore, heterogeneity of treatment effects will lead to funnel plot asymmetry if the true treatment effect is larger

* Corresponding author. Tel.: 44 117 928 7396; Fax: 44 117 928 7325. E-mail address: jonathan.sterne@bristol.ac.uk (J.A.C. Sterne).

in the smaller trials [6,8]. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome which is affected by the intervention [9]. Trials conducted in high-risk patients will also tend to be smaller, because of the difficulty in recruiting such patients. Funnel plots were first used in educational research and psychology [1], mainly for meta-analyses of continuous outcome variables in which standardized mean difference was plotted against sample size. In medical research vertical axes based on the standard error or variance of the treatment effect estimate have been increasingly used. A majority of trials in medicine have binary outcomes, and treatment effects are usually expressed as risk or odds ratios, although risk differences may also be used to measure treatment effects. Meta-analysts thus face a wide array of choices for both vertical and horizontal axes in funnel plots. This leads to the danger that the funnel plot chosen for a particular metaanalysis may be that which best conveys the message desired by the investigator, or may not be appropriate for detecting bias [10]. The purpose of this article is to provide guidelines for the choice of axes in funnel plots of metaanalyses with binary outcomes.

0895-4356/01/$ see front matter 2001 Elsevier Science Inc. All rights reserved. PII: S0895-4356(01)00 3 7 7 - 8

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

1047

2. Choice of vertical axis in funnel plots: case study The randomized controlled trials of magnesium treatment in the prevention of death following myocardial infarction (Table 1) are a well known example where publication bias, demonstrated by an asymmetrical funnel plot [5,6], has been suggested as an explanation for the discrepancy between meta-analyses which showed a clear beneficial effect of magnesium therapy on mortality [11,12] and a subsequent large trial which showed no effect [13]. Fig. 1 shows funnel plots for these 16 trials, using six different choices of vertical axis. In each case, the horizontal axis is the log odds ratio, and the vertical line shows the summary log odds ratio calculated using fixed effects meta-analysis. Calculation of the summary log odds ratio using fixed rather than random-effects meta-analysis is preferable because the random-effects estimate gives greater relative weight to smaller studies, and will therefore be more affected if publication bias is present [14]. Fig. 1A uses standard error (SE) as the vertical axis. The largest studies have the smallest standard errors, so to place the largest trials at the top of the graph, the axis has to be inverted (standard error 0 at the top). The diagonal lines show the expected 95% confidence intervals around the summary estimate, i.e. [summary effect estimate (1.96 SE)] and [summary effect estimate (1.96 SE)] for each SE on the vertical axis. They indicate the extent of between-trial heterogeneity: in the absence of heterogeneity 95% of the trials should lie within the funnel defined by these straight lines. Fig. 1B uses precision (defined as 1/SE) as the vertical axis. No inversion of the axis is required, because the larger
Table 1 Mortality results from 16 trials of intravenous magnesium in acute myocardial infarction Magnesium Study (first author) Morton Rasmussen Smith Abraham Feldstedt Schechter Ceremuzynski Bertschat Singh Pereira Schechter & Hod 1 Golf Thgersen LIMIT-2 Schechter & Hod 2 ISIS-4 Year 1984 1986 1986 1987 1988 1989 1989 1989 1990 1990 1991 1991 1991 1992 1995 1995 Deaths 1 9 2 1 10 1 1 0 6 1 2 5 4 90 4 2216 Total no. of patients 40 135 200 48 150 59 25 22 76 27 89 23 130 1159 107 29011 Control Deaths 2 23 7 1 8 9 3 1 11 7 12 13 8 118 17 2103 Total no. of patients 36 135 200 46 148 56 23 21 75 27 80 33 122 1157 108 29039

the study the smaller the standard error and hence the larger the precision. The lines showing expected 95% confidence intervals are derived by calculating the standard error corresponding to the value of the precision on the vertical axis. They are sharply curved and the plot emphasises the difference between the largest studies while smaller studies are compressed at the bottom. Fig. 1C and D use variance ( SE2) and inverse variance (1/variance) respectively for the vertical axis. In Fig. 1C, the smaller studies are further apart and the larger studies closer together than for standard error, so that the funnel is flattened at the top. Fig. 1D might be thought informative since the weights in fixed-effects metaanalysis are proportional to the inverse variance. However this plot compresses the smaller studies together even more than Fig. 1B, so that the expected 95% confidence intervals are even more sharply curved. Fig. 1E and F use sample size and log sample size, respectively, for the vertical axis. While Fig. 1F gives a similar impression of asymmetry as Fig. 1A, Fig. 1E compresses the smaller studies at the bottom to an even greater degree than Fig. 1B and in a similar manner to Fig. 1D. Because the standard error of the log odds ratio depends on the number of events in each group as well as on the number of patients, it is not possible to calculate expected 95% confidence intervals. The magnesium trials are unusual in that they include an extremely large trial, the Fourth International Study of Infarct Survival (ISIS-4) which enrolled 58,040 patients [13]. This trial is omitted from the funnel plots in Fig. 2. However Fig. 2B, using precision as the vertical axis, still emphasises the differences between the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2) [15] (2316 patients) and the remaining trials, and the lines denoting expected 95% confidence limits are still curved. The funnel in Fig. 2C is again rather flattened at its top, while Fig. 2D and E again compress all the other studies at the bottom of the plot. Even when the LIMIT-2 trial is also omitted, the 95% confidence lines for the funnel plot using precision as the vertical axis remain curved (Fig. 3). 3. Choice of horizontal axis in funnel plots: empirical study The choice of treatment effect measure may affect the interpretation of randomized trials and meta-analyses [16]. In practice, most meta-analyses use ratio measures of treatment effect (odds ratio or relative risk), although risk differences are sometimes also used. We conducted an empirical study to investigate whether the prevalence of funnel plot asymmetry in published meta-analyses depends on the choice of treatment effect. As described in detail elsewhere [8], we hand searched volumes 199397 of four general medicine and four specialist journals for meta-analyses based on at least five trials with binary endpoints which randomized individuals and reported the number of patients and events in each trial. We

ISIS-4: Fourth International Study of Infarct Survival. 0.5 was added to all cells in the 2 2 table to allow calculation of the odds ratio for the Bertschart trial. Bibliographic references are available from the authors on request.

1048

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

Fig. 1. Funnel plots for the magnesium trials with different vertical axes. The points correspond to the treatment effects from individual trials, and the diagonal or curved lines show the expected 95% confidence intervals around the summary estimate. Odds ratios are plotted on a logarithmic scale, to ensure that effects of the same magnitude but opposite directions are equidistant from 1.0.

found 78 meta-analyses, containing a total of 1145 trials, whose characteristics are shown in Table 2. When we performed standard chi-square tests of heterogeneity using the odds ratio, risk ratio and risk difference as outcome measures, the proportions of the 78 meta-analyses with heterogeneity P-value less than 0.05 were 0.29, 0.27 and 0.35, respectively. For each meta-analysis, we investigated the evidence for funnel plot asymmetry by doing the analysis proposed by Egger et al. [6]. As explained in more detail elsewhere [8], this is algebraically identical to a regression (weighted by the inverse variance of the treatment effect) of treatment effect (log odds ratio, log risk ratio or risk difference) on its standard error: treatment effect = b 0 + b 1 SE of treatment effect Evidence for funnel plot asymmetry is measured by the 0 then the estimated asymmetry coefficient b1: if b1

funnel plot is symmetric. Because of the low power of tests for funnel plot asymmetry [8], and consistent with previous work [6], we used P 0.1 in testing for asymmetry. We have shown elsewhere [8] that this regression test for funnel plot asymmetry is more powerful than the rank correlation method proposed by Begg and Mazumdar [17]. Analyses were done using the statistical package Stata (Stata Corporation, College Station, TX). Both the regression and rank correlation tests may be done using the Stata command metabias [18], which also draws funnel plots. There was evidence for funnel plot asymmetry for one or more of the treatment effect measures in 32 of the 78 metaanalyses. The number of meta-analyses with evidence of asymmetry was 21 (27%), 22 (28%) and 17 (22%) for log odds ratio, log risk ratio and risk difference, respectively. Fig. 4 is a Venn diagram showing the extent of agreement on the evidence for asymmetry between the three treatment effects. While there was substantial concordance between log odds ratio and log risk ratio, use of risk difference iden-

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

1049

Fig. 2. Funnel plots for the magnesium trials, excluding the fourth International Study of Infarct Survival (ISIS-4), with different vertical axes. The points correspond to the treatment effects from individual trials, and the diagonal or curved lines show the expected 95% confidence intervals around the summary estimate.

tified asymmetry in eight trials not identified using the other treatment effect measures.

4.1. Choice of vertical axis Clearly, the choice of variables used in funnel plots should be consistent with the fundamental assumption underlying the method: that in the absence of bias a plot from trials which estimate the same treatment effect will be symmetrical and bear some resemblance to a funnel. This is the case for standard error and variance (and functions thereof) but not necessarily for sample size or functions of sample size. Further, the major factor affecting the probability of publication of a study is its P-value [2,3,2123], which is a function of standard error or variance rather than of sample size. The power of a study, and the precision of the estimate of a treatments effect, is determined both by the total sample size and by the number of participants developing the event of interest. A study with 100,000 patients and 10 events is less powerful and will produce less precise estimates than a study with 1000 patients and 100 events. In other words, studies with very different sample sizes may have the same standard error and precision, and vice versa. In the absence

4. Discussion The potential for bias in the location, selection or conduct of the component studies included in meta-analyses is increasingly recognized. Funnel plots are a useful graphic means of checking whether small study effectsa tendency for treatment effect estimates in small studies to differ from those in larger studiesmay have distorted the results of a meta-analysis [19]. This could be due to publication bias, other reporting biases, low methodological quality of smaller studies or true heterogeneity in treatment effects [20]. However, there is confusion over how funnel plots should be constructed, and the choice of method may affect the shape of plots [10]. We addressed this issue in a case study and an empirical investigation of published meta-analyses.

1050

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

Fig. 3. Funnel plot for the magnesium trials, using precision as the vertical axis and excluding the fourth International Study of Infarct Survival (ISIS-4) and the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2). The points correspond to the treatment effects from individual trials, and the diagonal or curved lines show the expected 95% confidence intervals around the summary estimate. Fig. 4. Venn diagram showing evidence for funnel plot asymmetry in 78 published meta-analyses, when assessed using the odds ratio, risk ratio and risk difference as the measure of treatment effect.

of bias the shape of plots using sample size on the vertical axis will therefore not necessarily correspond to a funnel. A predictable shape in the absence of bias is required in order to use any plot to assess evidence for bias. For the same reason it is not possible to calculate expected 95% confidence limits for funnel plots of sample size. This is illustrated in Fig. 5, which shows funnel plots of the log odds ratio of endstage renal failure against standard error and sample size for a meta-analysis of 10 trials of ACE inhibitors in non-diabetic renal disease [24]. The event rate was low in the largest trial [25] so that the estimate of the treatment effect was less precise than the estimates from all except one of the smaller trials. Sample size and functions of sample size are therefore not a good choice for the vertical axis in funnel plots. Standard error, precision (the inverse of standard error), variance and the inverse of variance are four valid choices
Table 2 Characteristics of 78 meta-analyses published in eight journals 1993 to 1997 Characteristic Place of publication Annals of Internal Medicine BMJ JAMA Lancet Specialist journala No. of trials Median (range) Mean (SD) Outcome measure used in original report Odds ratio Relative risk Risk difference 10 (12.8) 25 (32.1) 13 (16.7) 13 (16.7) 17 (21.8) 10.5 (5 to 62) 14.7 (12.2) 53 (68.0) 21 (26.9) 4 (5.1)

Numbers (%) are shown unless otherwise stated a American Journal of Cardiology, Cancer, Circulation or Obstetrics and Gynecology.

for the vertical axis. However, there are important differences in the shape of plots. Our case study indicates that the use of standard error is likely to be preferable in many situations. Interpretation of funnel plots is facilitated by inclusion of lines representing the 95% confidence limits around the summary treatment effect which show the expected distribution of studies in the absence of heterogeneity or of selection biases. Such lines can be drawn for all four variables; however the lines will be straight and thus represent the shape of an inverted funnel for standard error only. For all other measures lines will be curved and, in the case of precision and inverse of variance, smaller studies will be compressed at the bottom of the graph. This makes the visual assessment of asymmetry more difficult and gives emphasis to the larger studies which on average are less prone to bias. Precision or the inverse of variance might be preferable in studies which focus on a comparison of meta-analyses of small trials with subsequent large trials [26]. Finally, funnel plots of standard error correspond to the regression test to detect asymmetry [6] and therefore allow the identification of trials which may be influential in the regression analysis. Tang and Liu [10] examined the evidence for funnel plot asymmetry in a sample of 198 meta-analyses published in the Cochrane Library using either precision or sample size on the vertical axis. They found that the shape of a funnel plot depends on the choice of the method to construct the plot but did not note that plots against sample size will not necessarily have a funnel shape. Tang and Liu used both the regression test proposed by Egger et al. [6] and a regression method of their own in which the covariate is the square root of the total sample size (rather than the standard error) and often obtained discrepant results depending on which

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

1051

Fig. 5. Funnel plots of the log odds ratio of end-stage renal failure against standard error (lower panel) and sample size (upper panel) for a meta-analysis of 10 trials of ACE inhibitors in non-diabetic renal disease [24]. The points correspond to the treatment effects from individual trials, horizontal lines indicate 95% confidence intervals and the diagonal lines show the expected 95% confidence intervals around the summary estimate. The rate of renal failure was low in the largest trial (indicated by an open circle) so that the estimate of the treatment effect was less precise than the estimates from all except one of the smaller trials.

statistical method was used. This is not surprising: the regression test proposed by Egger et al. [6] is based on the assumption that the process of selection for publication is driven by the statistical significance of the trials results, which will lead to an association between treatment effect and standard error. A recent study compared the power of different methods to detect publication bias, using simulations in which a studys probability of inclusion in a metaanalysis was based on its P-value [27]. This study concluded that the regression test proposed by Egger et al. [6]

had greater power than a regression against total sample size, although concerns were raised about the Type I error rate (see also the discussion in Sterne et al. [8]). Copas [28] performed sensitivity analyses in which the value of the estimated treatment effect is computed under a range of assumptions about the severity of the selection bias and showed that these can lead to an approximately linear association between treatment effect and its standard error [29]. There is also a considerable body of empirical evidence that statistically significant results are more likely to

1052

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

Fig. 6. Funnel plots for a meta-analysis of trials of the effect of ACE inhibitors on mortality in heart failure [38]. The points correspond to the log risk ratio or log odds ratio from individual trials, and the diagonal lines show the expected 95% confidence intervals around the summary estimate. For risk ratios, there is less asymmetry when the outcome is mortality (upper plot) that when the outcome is survival (middle plot). The lower plot uses odds ratios: the shape of funnel plots using odds ratios is the same whether the outcome is mortality or survival.

be published than non-significant ones: this includes studies comparing published trials with trials included in trial registries [30,31], investigations into the full publication of conference abstracts [32,33] and follow-up studies of proposals submitted to ethics committees [2,3,21,23] and of research funded by the National Institutes of Health [34]. The Tang and Liu approach [10] appears to postulate that

it is the square root of the total sample size of a trial which is most closely related to the probability of publication; an assumption which is supported by little empirical evidence. Indeed, a recent synthesis of the available data found that the sample size of a study was only weakly associated with the probability of publication whereas a consistent association was found with statistically significant results [22]. We

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055

1053

acknowledge, however, that in some situations other factors, such as the sample size, may be more strongly related to the probability of publication than the level of statistical significance. 4.2. Choice of horizontal axis We found that tests for funnel plot asymmetry reached different conclusions when treatment effects were expressed as risk differences than when ratio measures were used. Among the considerations when choosing the effect measure in meta-analyses of trials with binary outcomes are the measures consistency across trials and its ease of interpretation [35]. The less heterogeneous the statistic, the greater the justification for statistically combining the trials. In our sample of published meta-analyses we found that treatment effects measured using risk differences showed more between-trial heterogeneity than log odds ratio or log risk ratio. This is consistent with the findings of Deeks and Altman [35,36] in a study of almost 2000 meta-analyses of binary outcomes published on the Cochrane Library. Engels et al. [37] also found that more meta-analyses had heterogeneous risk differences than odds ratios, in a sample of 125 meta-analyses published between 1990 and 1996. They pointed out that use of risk differences can lead to trials with very few events in either treatment or control groups being given more weight than trials with many events [37]. This, and the greater heterogeneity of the risk difference between trials, will have contributed to the discrepant results of tests for funnel plot asymmetry in our study of 78 meta-analyses.

Fewer funnel plots were classified as asymmetric using risk differences (22%) than using odds ratios (27%) or risk ratios (28%): increased between-trial heterogeneity will reduce power to detect funnel plot asymmetry. Causes of between-trial heterogeneity may also lead to funnel plot asymmetry, if they are associated with study size. For example, if risk ratios were identical across trials then risk differences would by definition be larger in trials in which event rates were larger (i.e., trials in high-risk groups). In some meta-analyses trials in high-risk patients are smaller because high-risk patients are less common. This would lead to funnel plot asymmetry, not caused by bias, in funnel plots using risk differences. Given that there tends to be more heterogeneity using risk differences, these will not usually be the best choice of horizontal axis in funnel plots. Independent of the reason for funnel plot asymmetry, its presence calls into question the wisdom of statistically combining studies in meta-analysis. Conclusions from tests of funnel plot asymmetry were similar whether log odds ratio or log risk ratio was chosen. This could be expected as odds ratio and risk ratio give similar results unless event rates are high. Indeed, discrepancies between these measures arose only when the event rate was 30% in one or more trials. In this situation the odds ratio scale is preferable because it is not constrained (for example, a risk of 50% cannot increase by more than a factor of 2). Furthermore, funnel plots based on the odds ratio have the same shape whether the outcome is defined as occurrence or non-occurrence of the event of interest (for example, smoking cessation or continued smoking). In contrast,

Table 3 Possible choices of axis in funnel plots: advantages, disadvantages and recommendations Axis / measure Vertical axis Standard error Advantages and disadvantages Funnel shape with straight 95% confidence lines. Emphasis of the plot is on smaller studies where bias is more likely. Axis has to be inverted to place the largest trials at the top of the graph. Plot is not funnel shaped; 95% confidence lines are curved. Emphasis of the plot is on larger studies; smaller studies are compressed at the bottom. Plot is not funnel shaped; 95% confidence lines are curved. Emphasis of the plot is on smaller studies where bias is more likely. Plot is not funnel shaped; 95% confidence lines are curved. Emphasis of plot is on larger studies, smaller studies are compressed at the bottom. Recommendations The best choice in most cases

Precision (1/SE)

Variance (SE2) Inverse variance (1/variance)

An option in studies which focus on a comparison of meta-analyses of small trials with subsequent large trials Not recommended An option in studies which focus on a comparison of meta-analyses of small trials with subsequent large trials, but precision would usually be better. Invalid choice

Sample size or log sample size Horizontal axis Log odds ratio

Expected shape of plot in absence of bias is unpredictable.

Log risk ratio

Scale is not constrained and plots have the same shape whether the outcome is defined as occurrence or non-occurrence of the disease. Odds ratios may be misinterpreted as risk ratios. Readily understood measure. Scale is naturally constrained so that heterogeneity may be introduced if the event rate is high.

The best choice in most cases

Risk difference

Readily understood measure. Often associated with increased heterogeneity which may result in funnel plot asymmetry which is not apparent when ratio measures are used.

Valid choice in many cases but not recommended if the event rate is high. Can give different conclusions depending on outcome definition. Not recommended in most cases

1054

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055 [9] Davey Smith G, Egger M. Who benefits from medical interventions? Treating low risk patients can be a high risk strategy. Br Med J 1994; 308:724. [10] Tang JL, Liu JL. Misleading funnel plot for detection of bias in metaanalysis. J Clin Epidemiol 2000;53:47784. [11] Teo KK, Yusuf S, Collins R, Held PH, Peto R. Effects of intravenous magnesium in suspected acute myocardial infarction: overview of randomised trials. Br Med J 1991;303:1499503. [12] Horner SM. Efficacy of intravenous magnesium in acute myocardial infarction in reducing arrhythmias and mortality: meta-analysis of magnesium in acute myocardial infarction. Circulation 1992;86:7749. [13] ISIS-4 Collaborative Group. ISIS-4: A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet 1995;345:66987. [14] Poole C, Greenland S. Random-effects meta-analyses are not always conservative. Am J Epidemiol 1999;150:46975. [15] Woods KL, Fletcher S. Long term outcome after magnesium sulphate in suspected acute myocardial infarction: The Second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2). Lancet 1994;343: 8169. [16] Naylor CD, Chen E, Strauss B. Measured enthusiasm: does the method of reporting trial results alter perceptions of therapeutic effectiveness? Ann Intern Med 1992;117:91621. [17] Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994;50:108899. [18] Steichen TJ, Egger M, Sterne J. sbe19.1: Tests for publication bias in meta-analysis. Stata Technical Bulletin 1998;44:34. [19] Sterne JAC, Egger M, Smith DG. Investigating and dealing with publication and other biases. In: Egger M, Smith DG, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books, 2001. [20] Egger M, Davey Smith G. Meta-analysis: bias in location and selection of studies. Br Med J 1998;316:616. [21] Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Br Med J 1997; 315:6405. [22] Dickersin K. How important is publication bias? A synthesis of available data. AIDS Educ Prev 1997;9:1521. [23] Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:2816. [24] Giatras I, Lau J, Levey AS. Effect of angiotensin-converting enzyme inhibitors on the progression of nondiabetic renal disease: a metaanalysis of randomized trials. Ann Intern Med 1997;127:33745. [25] Himmelmann A, Hansson L, Hansson BG, Hedstrand H, Skogstrom K, Ohrvik J, Furangen A. ACE inhibition preserves renal function better than beta-blockade in the treatment of essential hypertension. Blood Press 1995;4:8590. [26] Cappelleri JC, Ioannidis JPA, Schmid CH, et al. Large trials vs metaanalysis of smaller trials. How do their results compare? JAMA 1996; 276:13328. [27] Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med 2001;20:64154. [28] Copas J. What works? Selectivity models and meta-analysis. J R Stat Soc A 1999;162:95109. [29] Copas JB, Shi JQ. Reanalysis of epidemiological evidence on lung cancer and passive smoking. Br Med J 2000;320:4178. [30] Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4:152941. [31] Stewart LA, Parmar MKB. Meta-analysis of the literaure or of individual patient data: is there a difference? Lancet 1993;341:41822. [32] De Bellefeuille C, Morrison CA, Tannock IF. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol 1992;3:18791. [33] Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts. A meta-analysis. JAMA 1994;272:15862.

the shape of funnel plots based on risk ratios will differ depending on the definition of the outcome as occurrence or non-occurrence. This is illustrated in Fig. 6 for a meta-analysis of trials of the effect of ACE inhibitors on mortality in heart failure [38]. Most trials had control group mortality rates below 10% but there were also trials with higher rates, including the Cooperative North Scandinavian Enalapril Survival Study (CONSENSUS) [39] with a control group mortality rate above 50%. When the outcome is mortality and treatment effects are measured using risk ratios (upper plot), the funnel plot seems moderately asymmetrical and the P-value from the regression test is 0.04. When the outcome is survival and treatment effects are again measured using risk ratios (middle plot), the plot is more clearly asymmetrical, as is the evidence from the regression test (P 0.006). A funnel plot using odds ratios (lower plot) looks similar to the plot using risk ratios and with mortality as the outcome, but the shape of this plot would be unchanged if the outcome were survival (the plot would simply be the mirror image, transposed around 0 on the y-axis). In this meta-analysis the most powerful studies were those which had the highest mortality rates, and so the extent to which the risk ratio is constrained is related to the position of the study in the funnel plot. Our recommendations for the choice of axis in funnel plots are summarized in Table 3. Although our understanding of the selection process leading to publication or nonpublication of research is imperfect, standard error is likely to be the best choice of the vertical axis in funnel plots, unless the aim is to emphasize the difference between the largest trials and all others, when precision is more appropriate. A log ratio measure, preferably the log odds ratio, should generally be used for the horizontal axis. Funnel plots are a useful graphic method but their interpretation is subjective. Standardizing the choice of axes should contribute to making them a more reliable tool in meta-analytic research. References
[1] Light RJ, Pillemer DB. Summing up. The science of reviewing research. Cambridge, MA: Harvard University Press, 1984. [2] Easterbrook PJ, Berlin J, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:86772. [3] Dickersin K, Min YL, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 1992;267:3748. [4] Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R Stat Soc A 1988;151:41963. [5] Egger M, Davey Smith G. Misleading meta-analysis. Lessons from an effective, safe, simple intervention that wasnt. Br Med J 1995; 310:7524. [6] Egger M, Davey Smith G, Schneider M, Minder CE. Bias in meta-analysis detected by a simple, graphical test. Br Med J 1997;315:62934. [7] Schulz KF, Chalmers I, Hayes RJ, Altman D. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:40812. [8] Sterne JAC, Gavaghan DJ, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000;53:111929.

J.A.C. Sterne, M. Egger / Journal of Clinical Epidemiology 54 (2001) 10461055 [34] Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clini Trials 1993;Doc No 50. [35] Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In: Egger M, Davey Smith G, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books, 2001. [36] Deeks JJ, Altman DG. Choosing an appropriate dichotomous effect measure for meta-analysis: empirical evidence of the appropriateness of the odds ratio and relative risk. Control Clin Trials 1997;18(Suppl. 3):84S5S.

1055

[37] Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Stat Med 2000;19:170728. [38] Garg R, Yusuf S, for the Collaborative Group on ACE Inhibitor Trials. Overview of randomised trials of angiotensin-converting enzyme inhibitors on mortality and morbidity in patients with heart failure. JAMA 1995;273:14506. [39] The CONSENSUS Trial Study Group. Effects of enalapril on mortality in severe congestive heart failure. N Engl J Med 1987;316:142935.