The uses and abuses of systematic reviews

Abstract: The uses and abuses of systematic reviews: the case of ivermectin in Covid-19 *Edmund J Fordham PhD FInstP EurIng Theresa A Lawrie MBBCh PhD Katherine MacGilchrist MSc EbMCsquared, a Community Interest Company Northgate House, Upper Borough Walls, Bath BA1 1RG, UK Andrew Bryant MSc Population Health Sciences Institute, Newcastle University Baddiley-Clark Building, Richardson Road, Newcastle upon Tyne NE2 4AX, UK *Corresponding author. Address: EbMCsquared CIC Northgate House, Upper Borough Walls, Bath BA1 1RG, UK Word count: 2839 words excl Refs 2 Tables; 2 Figures 55 references 1 Background A recent Cochrane review1 of ivermectin for the treatment and prevention of Covid-19 stands in contrast to the world-first PRISMA guidelines2 systematic review and meta-analysis by Bryant et al.3 (hereinafter “Bryant”) , which is criticised in the review with further commentary elsewhere4. We review here the approach of Popp et al.1 (hereinafter “Popp”). Ivermectin5, a WHO Essential Medicine6 of negligible cost7 and outstanding safety record8-13 has attracted significant attention for its potential use both in treatment and in the prevention of Covid-19, from its known anti-viral effect over a wide class of RNA viruses14 and specific action15 in vitro against SARS-CoV-2. These initial reports were bolstered by multiple reports16 of efficacy at all stages of the Covid-19 illness (from prophylaxis through to critical care17), in vitro18 and in silico19,20,21 studies of its likely modes of action, clinical experience reports from multiple contexts22, population-level case studies23, retrospective observational studies24, through to randomised controlled clinical trials3 (RCTs). These reports occupy all levels of the so-called “Evidence Quality Pyramid” in the doctrines of Evidence-Based Medicine. Systematic Reviews (SRs) and Meta-Analyses (MAs) occupy, in theory, the summit of the pyramid, and are considered the premier quality of evidence by Health Regulatory Authorities (HRAs) and for clinical practice guidelines by national and international health services. However, though SRs and MAs are carried out to identify, qualify or avoid the worst aspects of reporting bias, excluding all other evidence can become itself a powerful source of bias. The Cochrane organisation25 is dedicated to the development of a library of SRs, confined to the evidence. Cochrane SRs are traditionally conducted by volunteer reviewers, reducing the potential for financial conflicts of interest. Advocacy is not properly part of a SR but of processes such as DECIDE26 in which other stakeholders including public health professionals and patients should be involved. Cochrane has a tradition of accuracy, objectivity and attention to reliability through systematised but necessarily subjective criteria. Similar criteria are more generally available through the guidelines of the PRISMA Working Group2, followed in Bryant3. 2 However, SR methodologies are capable of abuse in respect of (i) choice of inclusion criteria (ii) choice of outcome measures, comparisons and analyses (iii) the subjectivity inevitable in Risk of Bias assessments, even when codified procedures and criteria are observed. We analyse these choices in Popp1. The Cochrane Review of Popp1: synopsis The stated objectives are to assess efficacy and safety of ivermectin compared to: no treatment, placebo, “standard of care” (SoC) or any “proven” intervention. The analyses compare hospitalised patients and..