Skip to main content
SpringerLink
Log in

Small-Study Effects in Meta-Analysis

  • Chapter
Meta-Analysis with R

Part of the book series: Use R! ((USE R))

Abstract

This chapter describes small-study effects in meta-analysis and how the issues they raise may be addressed. “Small-study effects” is a generic term for the phenomenon that smaller studies sometimes show different, often larger, treatment effects than large ones. This notion was coined by Sterne et al. [55]. One possible, probably the most well-known, reason is publication bias. This is said to occur when the chance of a smaller study being published is increased if it shows a stronger effect [3, 41, 52]. This can happen for a number of reasons, for example authors may be more likely to submit studies with “significant” results for publication or journals may be more likely to publish smaller studies if they have “significant” results. If this occurs, it in turn biases the results of meta-analyses and systematic reviews. There are a number of other possible reasons for small-study effects. One is selective reporting of the most favourable outcomes, known as outcome selection bias or outcome reporting bias [8, 9, 18, 61]. Another possible cause of small-study effects is clinical heterogeneity between patients in large and small studies; e.g., patients in smaller studies may have been selected so that a favourable outcome of the experimental treatment may be expected. In the case of a binary outcome, also a mathematical artefact arises from the fact that for the odds ratio or the risk ratio, the variance of the treatment effect estimate is not independent of the estimate itself [47]. This problem will be discussed in Sect. 5.2.2. Lastly, it can never be ruled out that small-study effects result from mere coincidence [42]. Empirical studies have established evidence for these and other kinds of bias [19, 42, 53]. There is a vast range of tests for small-study effects [4, 20, 24, 38, 43, 48], most of them based on a funnel plot which will be introduced in Sect. 5.1.1.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    R code to generate the funnel plot is given in the web-appendix.

  2. 2.

    Alternatively, the command ms1.asd <- metabin(Ee, Ne, Ec, Nc, data=data11, sm="ASD") could have been used.

References

  1. A’hern, R.P.: Widening eligibility to phase II trials: constant arcsine difference phase II trials. Control. Clin. Trials 25, 251–264 (2004)

    Article  Google Scholar 

  2. Armitage, P., Berry, G., Matthews, J.N.S.: Statistical Methods in Medical Research. Blackwell Science, Oxford (1987)

    Google Scholar 

  3. Begg, C.B., Berlin, J.A.: Publication bias: a problem in interpreting medical data. J. R. Stat. Soc. Ser. A 151, 419–445 (1988). (C/R: p445-463)

    Google Scholar 

  4. Begg, C.B., Mazumdar, M.: Operating characteristics of a rank correlation test for publication bias. Biometrics 50, 1088–1101 (1994)

    Article  MATH  Google Scholar 

  5. Berkey, C.S., Hoaglin, D.C., Mosteller, F., Colditz, G.A.: A random-effects regression model for meta-analysis. Stat. Med. 14, 395–411 (1995)

    Article  Google Scholar 

  6. Carpenter, J., Rücker, G., Schwarzer, G.: copas: an R package for fitting the Copas selection model. R J. 1(2), 31–36 (2009)

    Google Scholar 

  7. Carpenter, J.R., Schwarzer, G., Rücker, G., Künstler, R.: Empirical evaluation showed that the Copas selection model provided a useful summary in 80 % of meta-analyses. J. Clin. Epidemiol. 62, 624–631 (2009)

    Article  Google Scholar 

  8. Chan, A.W., Hróbartsson, A., Haahr, M.T., Gøtzsche, P.C., Altman, D.G.: Empirical evidence for selective reporting of outcomes in randomized trials. comparison of protocols to published articles. J. Am. Med. Assoc. 291, 2457–2465 (2004)

    Google Scholar 

  9. Chan, A.W., Krleza-Jeric, K., Schmid, I., Altman, D.G.: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Can. Med. Assoc. J. 171, 735–740 (2004)

    Article  Google Scholar 

  10. Copas, J.: What works?: Selectivity models and meta-analysis. J. R. Stat. Soc. Ser. A 162, 95–109 (1999)

    Article  Google Scholar 

  11. Copas, J., Lozada-Can, C.: The radial plot in meta-analysis: approximations and applications. Appl. Stat. 58(3), 329–344 (2009)

    MathSciNet  Google Scholar 

  12. Copas, J.B., Malley, P.F.: A robust p-value for treatment effect in meta-analysis with publication bias. Stat. Med. 27(21), 4267–4278 (2008)

    Article  MathSciNet  Google Scholar 

  13. Copas, J., Shi, J.Q.: Meta-analysis, funnel plots and sensitivity analysis. Biostatistics 1, 247–262 (2000)

    Article  MATH  Google Scholar 

  14. Copas, J.B., Shi, J.Q.: A sensitivity analysis for publication bias in systematic reviews. Stat. Methods Med. Res. 10, 251–265 (2001)

    Article  MATH  Google Scholar 

  15. Duncan, B., Olkin, I.: Bias of estimates of the number needed to treat. Stat. Med. 24, 1837–1848 (2005)

    Article  MathSciNet  Google Scholar 

  16. Duval, S., Tweedie, R.: A nonparametric “Trim and Fill” method of accounting for publication bias in meta-analysis. J. Am. Stat. Assoc. 95, 89–98 (2000)

    MATH  MathSciNet  Google Scholar 

  17. Duval, S., Tweedie, R.: Trim and Fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, 455–463 (2000)

    Article  MATH  Google Scholar 

  18. Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A., Cronin, E., Decullier, E., Easterbrook, P.J., Elm, E.V., Gamble, C., Ghersi, D., Ioannidis, J.P., Simes, J., Williamson, P.R.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3(8), e3081 (2008). Doi: 10.1371/journal.pone.0003081

    Article  Google Scholar 

  19. Easterbrook, P.J., Berlin, J.A., Gopalan, R., Matthews, D.R.: Publication bias in clinical research. Lancet 337, 867–872 (1991)

    Article  Google Scholar 

  20. Egger, M., Smith, G.D., Schneider, M., Minder, C.: Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315, 629–634 (1997)

    Article  Google Scholar 

  21. Galbraith, R.F.: A note on graphical presentation of estimated odds ratios from several clinical trials. Stat. Med. 7, 889–894 (1988)

    Article  Google Scholar 

  22. Galbraith, R.F.: Graphical display of estimates having differing standard errors. Technometrics 30, 271–281 (1988)

    Article  Google Scholar 

  23. Gleser, L.J., Olkin, I.: Models for estimating the number of unpublished studies. Stat. Med. 15, 2493–2507 (1996)

    Article  Google Scholar 

  24. Harbord, R.M., Egger, M., Sterne, J.A.: A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat. Med. 25(20), 3443–3457 (2006)

    Article  MathSciNet  Google Scholar 

  25. Higgins, J.P., Green, S. (eds.): Cochrane Handbook for Systematic Reviews of Interventions—Version 5.1.0 [updated March 2011]. The Cochrane Collaboration (2011). URL http://www.cochrane-handbook.org

  26. Ioannidis, J.P., Trikalinos, T.A.: An exploratory test for an excess of significant findings. Clin. Trials 4(3), 245–253 (2007)

    Article  MathSciNet  Google Scholar 

  27. Ioannidis, J.P.A., Trikalinos, T.A.: The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Can. Med. Assoc. J. 176(8), 1091–1096 (2007)

    Article  Google Scholar 

  28. Kasuya, E.: Angular transformation—another effect of different sample sizes. Ecol. Res. 19, 165–167 (2004)

    Article  Google Scholar 

  29. Kendall, M., Gibbons, J.D.: Rank Correlation Methods, 5th edn. Edward Arnold, London (1990)

    MATH  Google Scholar 

  30. Lau, J., Ioannidis, J.P.A., Terrin, N., Schmid, C.H., Olkin, I.: The case of the misleading funnel plot. Br. Med. J. 333, 597–600 (2006)

    Article  Google Scholar 

  31. Light, R.J., Pillemer, D.B.: Summing up. In: The Science of Reviewing Research. Harvard University Press, Cambridge, MA (1984)

    Google Scholar 

  32. Macaskill, P., Walter, S.D., Irwig, L.: A comparison of methods to detect publication bias in meta-analysis. Stat. Med. 20, 641–654 (2001)

    Article  Google Scholar 

  33. McCullagh, P., Nelder, J.: Generalized Linear Models. Chapman & Hall, London (1989)

    Book  MATH  Google Scholar 

  34. Moore, R.A., Tramer, M.R., Carroll, D., Wiffen, P.J., McQuay, H.J.: Quantitive systematic review of topically applied non-steroidal anti-inflammatory drugs. Br. Med. J. 316(7128), 333–338 (1998)

    Article  Google Scholar 

  35. Moreno, S., Sutton, A., Ades, A., Stanley, T., Abrams, K., Peters, J., Cooper, N.: Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med. Res. Methodol. 9, 2 (2009). URL http://www.biomedcentral.com/1471-2288/9/2/abstract

  36. Moreno, S.G., Sutton, A.J., Turner, E.H., Abrams, K.R., Cooper, N.J., Palmer, T.P., Ades, A.E.: Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. Br. Med. J. 339, b2981 (2009). Doi: 10.1136/bmj.b2981

    Article  Google Scholar 

  37. Orwin, R.G.: A fail-safe N for effect size in meta-analysis. J. Educ. Stat. 8, 157–159 (1983)

    Article  Google Scholar 

  38. Peters, J.L., Sutton, A.J., Jones, D.R., Abrams, K.R., Rushton, L.: Comparison of two methods to detect publication bias in meta-analysis. J. Am. Med. Assoc. 295, 676–680 (2006)

    Article  Google Scholar 

  39. Peters, J.L., Sutton, A.J., Jones, D.R., Abrams, K.R., Rushton, L.: Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat. Med. 27, 4544–4562 (2007)

    Article  MathSciNet  Google Scholar 

  40. Peters, J.L., Sutton, A.J., Jones, D.J., Abrams, K.R., Rushton, L.: Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J. Clin. Epidemiol. 61(10), 991–996 (2008)

    Article  Google Scholar 

  41. Rosenthal, R.: The “file drawer problem” and tolerance for null results. Psychol. Bull. 86, 638–641 (1979)

    Article  Google Scholar 

  42. Rothstein, H.R., Sutton, A.J., Borenstein, M.: Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Wiley, Chichester (2005)

    Book  MATH  Google Scholar 

  43. Rücker, G., Schwarzer, G., Carpenter, J.R.: Arcsine test for publication bias in meta-analyses with binary outcomes. Stat. Med. 27(5), 746–763 (2008)

    Article  MathSciNet  Google Scholar 

  44. Rücker, G., Schwarzer, G., Carpenter, J., Olkin, I.: Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Stat. Med. 28(5), 721–738 (2009)

    Google Scholar 

  45. Rücker, G., Schwarzer, G., Carpenter, J., Binder, H., Schumacher, M.: Treatment effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics 12(1), 122–142 (2010). Doi:10.1136/jme.2008.024521.

    Article  Google Scholar 

  46. Rücker, G., Carpenter, J., Schwarzer, G.: Detecting and adjusting for small-study effects in meta-analysis. Biom. J. 53(2), 351–368 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  47. Schwarzer, G., Antes, G., Schumacher, M.: Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes. Stat. Med. 21, 2465–2477 (2002)

    Article  Google Scholar 

  48. Schwarzer, G., Antes, G., Schumacher, M.: A test for publication bias in meta-analysis with sparse binary data. Stat. Med. 26, 721–733 (2007)

    Article  MathSciNet  Google Scholar 

  49. Schwarzer, G., Carpenter, J.R., Rücker, G.: Empirical evaluation suggests Copas selection model preferable to trim-and-fill method for selection bias in meta-analysis. J. Clin. Epidemiol. 63, 282–288 (2010)

    Article  Google Scholar 

  50. Schwarzer, G., Carpenter, J., Rücker, G.: metasens: Advanced statistical methods to model and adjust for bias in meta-analysis (2014). URL http://cran.R-project.org/package=metasens. R package version 0.1-0

  51. Stanley, T.D.: Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf. Bull. Econ. Stat. 70(105–127) (2008)

    Google Scholar 

  52. Sterling, T.D.: Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J. Am. Stat. Assoc. 54, 30–34 (1959). (Comment: V54 p593)

    Google Scholar 

  53. Sterling, T.D., Rosenbaum, W.L., Weinkam, J.J.: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am. Stat. 49, 108–112 (1995)

    Google Scholar 

  54. Sterne, J.A.C., Egger, M.: Funnel plots for detecting bias in meta-analysis: guideline on choice of axis. J. Clin. Epidemiol. 54, 1046–1055 (2001)

    Article  Google Scholar 

  55. Sterne, J.A.C., Gavaghan, D., Egger, M.: Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J. Clin. Epidemiol. 53, 1119–1129 (2000)

    Article  Google Scholar 

  56. Sterne, J.A.C., Sutton, A.J., Ioannidis, J.P.A., Terrin, N., Jones, D.R., Lau, J., Carpenter, J., Rücker, G., Harbord, R.M., Schmid, C.H., Tetzlaff, J., Deeks, J.J., Peters, J., Macaskill, P., Schwarzer, G., Duval, S., Altman, D.G., Moher, D., Higgins, J.P.T.: Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Br. Med. J. 343, d4002 (2011). URL http://bmj.com/cgi/content/full/bmj.d4002. Doi: 10.1136/bmj.d4002

  57. Terrin, N., Schmid, C.H., Lau, J., Olkin, I.: Adjusting for publication bias in the presence of heterogeneity. Stat. Med. 22, 2113–2126 (2003)

    Article  Google Scholar 

  58. Terrin, N., Schmid, C.H., Lau, J.: In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J. Clin. Epidemiol. 58(9), 894–901 (2005)

    Article  Google Scholar 

  59. Thompson, S.G., Sharp, S.J.: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat. Med. 18, 2693–2708 (1999)

    Article  Google Scholar 

  60. Vevea, J.L., Hedges, L.V.: A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995)

    Article  MATH  Google Scholar 

  61. Williamson, P.R., Gamble, C.: Identification and impact of outcome selection bias in meta-analysis. Stat. Med. 24(10), 1547–1561 (2005)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Medical Biometry and Statistics, Medical Center – University of Freiburg, Freiburg, Germany

    Guido Schwarzer & Gerta Rücker

  2. MRC Clinical Trials Unit, London and London School of Hygiene and Tropical Medicine, London, UK

    James R. Carpenter

Authors
  1. Guido Schwarzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James R. Carpenter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerta Rücker
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Schwarzer, G., Carpenter, J.R., Rücker, G. (2015). Small-Study Effects in Meta-Analysis. In: Meta-Analysis with R. Use R!. Springer, Cham. https://doi.org/10.1007/978-3-319-21416-0_5

Download citation

Publish with us

Policies and ethics

Search

Navigation