Original articleFunnel plots for detecting bias in meta-analysis: Guidelines on choice of axis
Introduction
Funnel plots—scatter plots in which the treatment effects estimated from individual studies on the horizontal axis are plotted against a measure of study precision on the vertical axis—have been proposed as a means of detecting publication bias in meta-analysis [1]. In the absence of bias the graph resembles a symmetrical inverted funnel because the treatment effect estimates from smaller studies scatter more widely at the bottom of the graph, with the spread narrowing with increasing precision among larger studies. If there is publication bias because smaller studies which show no statistically significant effects remain unpublished 2, 3, then the funnel plot will appear asymmetrical 4, 5. Funnel plot asymmetry cannot, however, be interpreted as proof of publication bias in meta-analysis [6]. Asymmetry could also result from the overestimation of treatment effects in smaller studies of inadequate methodological quality [7]. Furthermore, heterogeneity of treatment effects will lead to funnel plot asymmetry if the true treatment effect is larger in the smaller trials 6, 8. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome which is affected by the intervention [9]. Trials conducted in high-risk patients will also tend to be smaller, because of the difficulty in recruiting such patients.
Funnel plots were first used in educational research and psychology [1], mainly for meta-analyses of continuous outcome variables in which standardized mean difference was plotted against sample size. In medical research vertical axes based on the standard error or variance of the treatment effect estimate have been increasingly used. A majority of trials in medicine have binary outcomes, and treatment effects are usually expressed as risk or odds ratios, although risk differences may also be used to measure treatment effects.
Meta-analysts thus face a wide array of choices for both vertical and horizontal axes in funnel plots. This leads to the danger that the funnel plot chosen for a particular meta-analysis may be that which best conveys the message desired by the investigator, or may not be appropriate for detecting bias [10]. The purpose of this article is to provide guidelines for the choice of axes in funnel plots of meta-analyses with binary outcomes.
Section snippets
Choice of vertical axis in funnel plots: case study
The randomized controlled trials of magnesium treatment in the prevention of death following myocardial infarction (Table 1) are a well known example where publication bias, demonstrated by an asymmetrical funnel plot 5, 6, has been suggested as an explanation for the discrepancy between meta-analyses which showed a clear beneficial effect of magnesium therapy on mortality 11, 12 and a subsequent large trial which showed no effect [13]. Fig. 1 shows funnel plots for these 16 trials, using six
Choice of horizontal axis in funnel plots: empirical study
The choice of treatment effect measure may affect the interpretation of randomized trials and meta-analyses [16]. In practice, most meta-analyses use ratio measures of treatment effect (odds ratio or relative risk), although risk differences are sometimes also used. We conducted an empirical study to investigate whether the prevalence of funnel plot asymmetry in published meta-analyses depends on the choice of treatment effect.
As described in detail elsewhere [8], we hand searched volumes
Discussion
The potential for bias in the location, selection or conduct of the component studies included in meta-analyses is increasingly recognized. Funnel plots are a useful graphic means of checking whether “small study effects”—a tendency for treatment effect estimates in small studies to differ from those in larger studies—may have distorted the results of a meta-analysis [19]. This could be due to publication bias, other reporting biases, low methodological quality of smaller studies or true
References (39)
- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Publication and related bias in meta-analysispower of statistical tests and prevalence in the literature
J Clin Epidemiol
(2000) - et al.
Misleading funnel plot for detection of bias in meta-analysis
J Clin Epidemiol
(2000) - et al.
Long term outcome after magnesium sulphate in suspected acute myocardial infarctionThe Second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2)
Lancet
(1994) - et al.
Meta-analysis of the literaure or of individual patient datais there a difference?
Lancet
(1993) - et al.
The fate of abstracts submitted to a cancer meetingfactors which influence presentation and subsequent publication
Ann Oncol
(1992) - et al.
Summing up. The science of reviewing research
(1984) - et al.
Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards
JAMA
(1992) - et al.
Publication biasa problem in interpreting medical data
J R Stat Soc A
(1988) - et al.
Misleading meta-analysis. Lessons from “an effective, safe, simple” intervention that wasn't
Br Med J
(1995)
Bias in meta-analysis detected by a simple, graphical test
Br Med J
Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials
JAMA
Who benefits from medical interventions? Treating low risk patients can be a high risk strategy
Br Med J
Effects of intravenous magnesium in suspected acute myocardial infarctionoverview of randomised trials
Br Med J
Efficacy of intravenous magnesium in acute myocardial infarction in reducing arrhythmias and mortalitymeta-analysis of magnesium in acute myocardial infarction
Circulation
ISIS-4A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction
Lancet
Random-effects meta-analyses are not always conservative
Am J Epidemiol
Measured enthusiasmdoes the method of reporting trial results alter perceptions of therapeutic effectiveness?
Ann Intern Med
Operating characteristics of a rank correlation test for publication bias
Biometrics
Cited by (2677)
NSAIDs and Corticosteroids for the Postoperative Management of Age-Related Cataract Surgery: A Systematic Review and Meta-analysis
2024, American Journal of OphthalmologyA Review of Meta-Analyses in Plastic Surgery: Need for Adequate Assessment of Publication Bias
2024, Journal of Surgical ResearchArginine metabolomics in mood disorders
2024, HeliyonPrevalence of undernutrition among migrant, refugee, internally displaced children and children of migrated parents in lower-middle-income countries: A meta-analysis of published studies from last twelve years
2024, Diabetes and Metabolic Syndrome: Clinical Research and Reviews