gms | German Medical Science

Background/research question: Randomised controlled trials (RCTs) are the ideal study design to infer causality, but the most common evidence source in nutrition research are cohort studies. Since cohort studies provide potentially less trustworthy evidence, this has lead to an ongoing debate in nutritional epidemiology [1].

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study [2].

Methods: We searched Medline, Epistemonikos and the CDSR for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies as "more or less identical", "similar but not identical", "broadly similar". Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements.

Results: We included 82 BoE-pairs from 51 systematic reviews. Dietary interventions/exposures focused mainly on micronutrients (62.2%). With regard to the PI/ECO similarity degree, 57 BoE-pairs (69.5%) were rated as "similar but not identical" and 15 (18.3%) as "broadly similar".

On average, the BoE from RCTs and cohort studies yielded similar effect estimates: For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99, 1.10, I²=59%, τ²=0.02, prediction interval (PI) 0.77, 1.41). For continuous outcomes, the pooled DSMD was -0.09 (95% CI -0.26, 0.09, PI -0.55, 0.38). However, the wide prediction intervals suggest that differences could be considerably larger or smaller in either direction. Subgroup analyses revealed that disagreement was driven by PI/ECO dissimilarity, in particular by differences in type of intake/exposure.

Conclusion: We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.

Competing interests: The authors have no relevant financial or non-financial interest to disclose.