gms | German Medical Science

Background/research question: Meta-analyses involving a small number of trials or rare events pose a challenge for statistical analysis. Evidence suggests that conventional two-stage statistical methods can lead to distorted results in these sparse data situations. Though better-performing one-stage methods have become available in recent years, these methods appear not sufficiently implemented and two-stage methods are still often used. The actual impact of using two-stage methods in practice, however, remains unknown. This study aims to quantify the impact by re-analysing meta-analyses from the Cochrane Database of Systematic Reviews (CDSR) in two sparse data situations: when meta-analyses included trials with zero events in one or both arms, or when meta-analyses contained only a few trials.

Methods: For each scenario, we computed one-stage statistical methods, namely the generalized linear mixed model (GLMM), the beta-binomial model (BBM) and the Bayesian binomial-normal hierarchical model using a weakly informative prior (BNHM-WIP). We then compared their impact on the results to the conventionally used two-stage methods, namely the Peto-Odds-Ratio (PETO) and DerSimonian-Laird method (DL) in case of zero event trials and DL, the Paule-Mandel (PM) and restricted maximum likelihood (REML) method in case of few trials.

Results: While all methods showed similar estimates for the pooled treatment effect, the results showed large variability in the statistical precision (length of CI and statistical significance) between methods. Specifically, two-stage methods in the zero event situation tended to estimate narrower CIs resulting in more significant meta-analyses than the one-stage methods. While differences between the two-stage and one-stage methods are less evident in the few trial situations, the one-stage methods proved less frequent statistically significant.

Conclusion: Our results confirm previous studies suggesting a high number of false-positive results in real meta-analyses in sparse data situations. In addition, the differences in the results provide evidence that method choice has a substantial impact on the outcome of meta-analyses and encourages the careful choice of an adequate method. In the situation of zero event trials, the BBM and BNHM-WIP appear to be promising candidates while using BBM, and additionally PM and REML for sensitivity analyses appears reasonable in case of few trials. Furthermore, Bayesian methods with carefully selected priors can be an alternative in the latter situation.

Competing interests: None of the authors has a financial conflict of interest. Oliver Kuss and Tim Mathes were involved in the development of using the BBM for meta-analyses involving few trials or zero events. The developers of the BNHM-WIP work at the same department as Tim Mathes and Maxi Schulz.