Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19

Abstract

Ivermectin has been identified as an inexpensive, readily available drug with the potential to be repurposed as a treatment for COVID-19, especially in countries with limited access to vaccines. Although multiple studies have been published in an attempt to evaluate its usefulness in COVID-19, many are small and not constructed appropriately to detect differences in important clinical outcomes (ie, death). For this reason, researchers have turned to meta-analyses to combine study results and draw summary conclusions regarding ivermectin's effectiveness. Two such meta-analyses recently published in the American Journal of Therapeutics concluded that ivermectin decreased mortality and improved other surrogate end points in COVID-19.1–4 A recently withdrawn article caused both authors to rework their meta-analyses without altering their main conclusions.1–5 We feel that shortcomings within both sets of meta-analyses and limitations in the component studies are significant enough to invalidate their main finding that ivermectin reduces mortality. A review of other meta-analyses on the same subject, containing many of the same individual studies, were similarly limited by poor design. KORY AND MARIK META-ANALYSES In their updated meta-analysis, the authors missed an opportunity to improve methodologic weaknesses within their original study.1,2 In both the original and follow-up meta-analyses, the authors did not list standard methodological items (eg, Preferred Reporting Items for Systematic Reviews-Metanalysis) and had no description of statistical techniques.6 Absence of these descriptors makes meta-analyses nonreproducible and nonreplicable, thereby limiting external validity.7 Their absence automatically classifies this meta-analysis as having critically low quality.8,9 The authors implied their follow-up was performed to exclude a retracted study. However, without explanation, the authors deleted a second study while adding 2 others.10 One newly added study was unblinded and a second enrolled patients with negative PCR tests and did not control for underling comorbid risks.11,12 In a larger omission, the authors did not include 3 double-blind placebo-controlled trials showing no effect on mortality—biasing results in favor of ivermectin.13–15 In their follow-up, the authors did not correct errors from the original meta-analysis that overstated ivermectin's mortality benefit. In their description of the study by Ravikirti et al, mortality was described as “improved” with ivermectin compared with controls “0% versus 6.9%, P = 0.019.”1,16 However, there was no statistical difference in mortality (0/55 vs. 4/57, Fisher exact P value = 0.12). A mortality benefit with ivermectin in severely ill patients was stated to be “of borderline statistical significance, 0% (0/11) versus 27.3% (6/22), P = 0.052” in the description of Hashim's study.1,17 This was incorrect, as the Fisher exact test, P value is 0.08. The phrases “borderline significant,” “approached” and “nearly statistically significant” used in this meta-analysis have been described as inappropriate and misleading as “results do not include movement and cannot approach significance because of the dichotomous definition” of statistically significant.1,18,19 Inclusion of unblinded, single-blinded, and nonrandomized studies limited the authors' ability to calculate reliable, reproducible, pooled estimates of treatment effects. The largest study within the reworked meta-analysis comprising 39% of all cases and 48% of all deaths was a chart review where only 3.5% of patients were given ivermectin.20 Patients also received multiple other treatments in an uncontrolled manner (eg, steroids, tocilizumab) with no comparison of disease severity or comorbidity between the groups. In the study by Khan et al, nonrandomization resulted in a sicker control population (46% control vs. 10% study oxygen requirement).21 Nonrandomization led to different treatment (39.8% dexamethasone in ivermectin vs. 19.6% in control group) in the retrospective study by Rajter et al.22 The authors inappropriately adjusted for baseline risk between the groups by constructing a prediction model using binary logistic regression that overfitted their data using 13 variables to create a model that predicted 53 outcomes.22 The ratio of studied prediction variables to outcome should be, at most, 1:10 or 1:20.23 In other studies, described as double-blind, randomized controlled trials (RCTs), study and control groups differed. In the study by Niaee et al,24 control patients had fewer positive PCR tests, lower oxygen saturations, higher body mass indices (BMI), and more initial CT scans compared with ivermectin patients. The authors miscalculated P values when comparing PCR positivity and initial CT performance between the groups. The values should be <0.001 for both comparisons and not 0.421, 0.527 as reported.24 Vital sign data appear to be in error as a diastolic blood pressure median and interquartile range of 80 mm Hg (80–80) was reported in 4 of 6 groups in the study. It is unlikely that half of all patients (60/120 patients) in the first 4 groups had the exact same diastolic blood pressure. In the study by Mahmud et al, 32 cases were lost to follow-up (LTFU).25 Study validity is affected when LTFU is >5% (9% in this study), LTFUs exceed the outcome (32 LTFUs vs. 3 deaths) or a worst-case scenario imputation alters results (ie, all study LTFU and no control LTFUs have adverse outcome).26–28 This imputation would result in 8.2% mortality in ivermectin cases versus 1.7% in controls (P < 0.01) and render the overall mortality difference between the groups nonsignificant [odds ratio, 0.48; 95% confidence interval (CI) 0.21–1.07, RevMan 5.4; MedCalc 19.7, Osteen Belgium] within this meta-analysis. BRYANT META-ANALYSES Bryant's meta-analyses corrected many methodological shortcomings within the first set of meta-analyses by including many Preferred Reporting Items for Systematic Reviews-Metanalysis items. The authors, however, did not appear to register their meta-analysis, a critical item that categorizes a meta-analysis as low quality.8,9 Study registration improves transparency and minimizes the risk of selective outcome reporting bias. Selection bias may have been introduced when authors of 2 previous pro-ivermectin (or pro ivermectin) meta-analyses were contacted to aid in identifying additional articles for this analysis. The authors did not list inclusion or exclusion criteria within their methods only stating that they searched for RCTs. Their final included articles were described as double blind, open label, and quasi-RCTs. Some experts believe quasiexperimental studies are not RCTs, thus, requiring exclusion because true randomization is absent.29,30 Many studies were undersized with no sample size calculations for any outcome in 6 studies with other studies calculating samples sizes to detect differences in surrogate end points: clinical recovery (n = 5), viral clearance (2), and CIs around combined group mortality (1).3,4 No study was constructed to detect mortality differences between ivermectin and control groups. Eleven studies were small (<100 patients) increasing the likelihood that findings are false and inflating the effects of treatments.31,32 Further misinterpretation of results can occur in undersized studies that make multiple comparisons without any statistical corrections. For example, Okumus et al and Shahbaznejad et al conducted 69 and 58 statistical comparisons in studies with only 69 and 60 total patients, respectively.11,12 Nine of 14 studies were described as having adequate random sequence generation and allocation concealment.3 Failed randomization/allocation occurred in 2 of these “adequate” studies with a different viral cycle threshold plus different polymerase chain reaction (PCR) positivity, BMI, and oxygen saturation between ivermectin and control groups.24,33 In 5 studies described as not having adequate allocation, one had different baseline SOFA scores and 4 did not analyze comorbidity or other treatments between the groups.11,13,14,17,34 These confounding biases can alter the results of individual studies, thus, altering results of the meta-analysis. The authors describe 6 of 14 studies as having an unclear or high risk for inadequate blinding.3 They state blinding is “less important” for evaluating evidence related to mortality.3 We disagree. Inadequate blinding can alter treatments, aggressiveness of care, and other actions taken to manage patients potentially resulting in different outcomes. Five studies lacked placebo controls,11,12,17,33,35 4 contained nonplacebo controls (ritonavir/lopinavir, doxycycline, chloroquine, hydroxychloroquine),12,13,33,35 and 3 added doxycycline to ivermectin treatment groups.17,25,36 Studies with active or nonplacebo controls, or that add drugs to treatment groups, do not allow ivermectin's effects to be isolated from those of the other drugs. Moreover, it is possible that active controls might worsen outcomes, falsely skewing results toward ivermectin's effectiveness. Several other aspects of studies within each set of meta-analysis limits the ability to draw summary conclusions. Variable ivermectin doses, duration of ivermectin use, admission status, variable testing or no COVID-19 testing, and timing of drug administration during the disease course obscure the interpretation of results. Mixing studies with varying disease severity definitions further limits the ability to make conclusions. Moreover, inclusion of 2 studies enrolling children was inappropriate because mortality differs dramatically between adults and children.12,17 In their original meta-analysis, Bryant et al3 performed sensitivity analyses to assess robustness of their results by performing trial sequential analysis to confirm that ivermectin reduced mortality. Reanalysis of the study by Niaee et al showed that it should have been recategorized as having a high risk of bias and excluded during sensitivity analysis.24 This study had data and statistical errors previously described and included PCR-negative patients with different between-group PCR positivity, BMI, oxygen saturation, and initial CT performance, indicating randomization failure. In our opinion, these issues require removal of this article from any meta-analysis. Importantly, this was the only article within the Bryant meta-analysis showing that ivermectin reduced mortality.4 Removal of this single article alters their results so that no significant difference now exists between ivermectin and control mortality (risk ratio, 0.73; 95% CI, 0.45–1.18). OTHER META-ANALYSES We performed a literature search (EMBASE, PubMed, Web of Science, bioRxiv, medRxiv, Google Scholar, January 12,020 to October 15, 2021) and found 18 additional meta-analyses (now 20 total) that evaluated ivermectin's effect on COVID-19 mortality (Table 1).37–54 There was substantial overlap of studies within meta-analyses. Fourteen of 18 had the majority of their included studies contained within the Bryant or Marik meta-analyses (Table 1). Eleven concluded that ivermectin prevented mortality (Table 1). Removing the withdrawn study by Elgazzar et al left 9 and removing the study by Niaee et al left only 5 meta-analyses concluding that ivermectin is effective (Table 1). These 5 include Marik's meta-analysis (previously critiqued), meta-analyses by Kow and Nardlli whose individual studies overlap completely with Marik's and Bryant's meta-analyses, a meta-analysis by Karale et al that found no mortality difference in subsets with RCTs or without active controls, and a 3-study meta-analysis by Padhy et al with the quality of evidence, Cochrane GRADE, rated as very low.2,4,5,24,44,46–48 Table 1. - Meta-analyses evaluating Ivermectin's association with mortality in patients with COVID-19. Author* publication date Mortality studies (without Elgazzar) Number (%) overlap with revised Bryant† Number (%) overlap with revised Marik† Ivermectin and overall mortality‡ Ivermectin and mortality excluding Elgazzar§ Ivermectin and mortality excluding Elgazzar/Niaee§ AMSTAR 2 meta-analysis quality¶ Siemeniuk51 July 30, 2020Living meta-analysis, updates periodically 8 (7) 7/7 (100%) 3/7 (43%) Network relative estimate0.31 (0.14 to 0.72) RR, 0.33 (0.14 to 0.77) RR, 0.54 (0.21 to 1.36) Mod/high Padhy48 November 23, 2020 3 0 2/3 (67%) OR, 0.53 (0.29 to 0.96) NA NA Critically low Kim45 December 30, 2020 2 0 2/2 (100%) Moderate/severeOR, 0.76 (0.25 to 2.33)Critical (one study)OR, 0.15 (0.04 to 0.57) NA NA Mod/high Castenada-Sabogal#,37 January 27, 2021 4 0 3 (75%) RR, 0.7 (0.31 to 2.28) NA NA Critically low Kow46 March 29, 2021 6 (5) 5/5 (100%) 5/5 (100%) OR, 0.21 (0.11 to 0.42) OR, 0.27 (0.14 to 0.55) OR, 0.37 (0.16 to 0.89) Critically low Nardelli47 May 8, 2021 7 (6) 6/6 (100%) 5/6 (83%) OR, 0.19 (0.1 to 0.34) OR, 0.27 (0.14 to 0.55) OR, 0.37 (0.16 to 0.89) Critically low Hariyanto41 June 6, 2021 8 (7) 7/7 (100%) 5/7 (71%) RR, 0.31 (0.15 to 0.62)** RR, 0.42 (0.24 to 0.74) RR, 0.55 (0.3 to 1) Mod/high Zein53 June 27, 2021 9 (8) 7/8 (88%) 5/8 (63%) RR, 0.39 (0.2 to 0.74) RR, 0.56 (0.34 to 0.93) RR, 0.6 (0.34 to 1.05) Low Roman50 June 28, 2021 5 5/5 (100%) 2/5 (40%) RR, 0.37 (0.12 to 1.13) NA RR, 0.62 (0.24 to 1.65) Critically low Hill42 July 6, 2021 11 (10) 9/10 (90%) 6/10 (60%) RR, 0.44 (0.25 to 0.77) RR, 0.57 (0.35 to 0.95) RR, 0.77 (0.51 to 1.16) Critically low Popp49 Jul 28, 2021 4 4/4 (100%) 1/4 (25%) Moderate–severe RR, 0.6 (0.14 to 2.51) mild to RR, 0.33 (0.01 to 8.05) NA NA Mod/high Izcovich43 August 21, 2021 12 (11) 9/11 (82%) 6/11 (55%) RR, 0.5 (0.28 to 0.88); RR, 0.96†† (0.58 to 1.96) RR, 0.64 (0.4 to 1.05) RR, 0.84 (0.57 to 1.24) Mod/high Bryant4 August 27, 2021 14 14/14 (100%) 6/14 (43%) RR, 0.51 (0.27 to 0.95) RR, 0.73 (0.45 to 1.18) Low Marik2 September 2, 2021 10 6/10 (60%) 10/10 (100%) OR, 0.39 (0.25 to 0.6) NA OR, 0.44 (0.28 to 0.71) Critically low Deng40 September 2, 2021 13 9/13 (69%) 4/13 (31%) OR 0.77 (0.5 to 1.19) NA NA Low Cruciani39 September 8, 2021 8 (7) 7/7 (100%) 4/7 (57%) RD to 0.02 (−0.05 to 0.01)‖ RD −0.01 (−0.04 to 0.01) NA Low Cheng38 September 16, 2021 3 2/3 (67%) 1/3 (33%) RR, 0.53 (0.11 to 2.61) NA NA Critically low Karale44 September 17, 2021 29 11/29 (38%) 10/29 (34%) OR, 0.54 (0.34 to 0.86) NA OR, 0.58 (0.36 to 0.92) Critically low Zhang‡‡,54 September 28, 2021 10 6/10 (60%) 4/10 (40%) OR, 0.61 (0.37 to 1) NA OR, 0.87 (0.51 to 1.49) Critically low Singh52 October 10, 2021 2 2/2 (100%) 2/2 (100%) OR, 0.45 (0.17 to 0.18) NA NA Low *Dates may be inaccurate when publication on preprint server changes to online or print journal publication. Within meta-analyses, listed first authors for included individual studies may have differed between preprint and final published articles (eg, Cepelowicz = Rajter, Fonseca = Bermijo Galan = Galan, Shahbazneiad = Rezai, Ravikirti = Kirti, Beltran-Gonzales = Gonzales).†Number (%) of individual studies within these meta-analyses that are also contained within Marik and Bryant revised meta-analyses.‡RD, RR, OR with 95% CIs in parenthesis.§Recalculated RD, OR, RR with deletion of Elgazzar study and with deletion of both Elgazzar/Niaee studies,5,24 Multiple meta-analyses have different total mortality and total cases for the same individual studies. When available, the mortality and total numbers used by each meta-analysis were used for each re-calculation. NA within columns indicatesnot applicable because Elgazzar or Niaee article is not contained within these meta-analyses. RevMan 5.4 and MedCalc 19.7, osteen Belgium were used to calculate values repeating the same statistical technique used in each study.¶Two reviewers (by consensus, disagreements to be settled by third reviewer–unnecessary) independently assessed each meta-analysis for the presence or absence of 7 critical items to classify its quality as critically low, low, or moderate/high.8,9 If all critical items are present, the meta-analysis is moderate or high quality. The absence of one item categorizes a meta-analysis as low quality and 2 or more items categorize an article as having critically low quality. (kappa = 0.72, 95% confidence interval, 0.49–0.94 for initial 2 reviewer agreement).‖Cruciani left off 4 deaths in control group, adding these back in changes initial RD to −0.03 (−0.06 to 0.01).#These authors treated the same article by Cepelowicz-Rajter [preprint (Cepelowicz) and final published article (Rajter)] as 2 separate studies when they were the same study. Thus, there were only 4 total studies and their initial calculations appear incorrect. Deleting the preprint version of the article alters their calculated OR to 0.68 (0.24–1.97).**The authors mislabeled and transposed their mortality and PCR Forest plots within their article. Figure 2 (a) and 2 (b) descriptions are transposed.††Excluding studies with a high risk of bias.‡‡The 10 included articles within this ivermectin mortality analysis were identified by comparing the ivermectin studies within Table S3 (Detailed Trial Characteristics) with those within Table S5 [Evaluation of risk of bias (mortality)].OR, odds ratio; RD, risk difference; RR, risk ratio (relative risk). A quality appraisal of all 20 meta-analyses using the AMSTAR-2 tool found that 15 had low or critically low quality.8,9 The 5 meta-analyses concluding that ivermectin was effective after article exclusions all had critically low quality2,44,46–48 (Table 1). Others have noted similar low quality in a large number of meta-analyses and recommended remediation by following best-practice guidelines for meta-analyses and using anonymized individual patient data (IPD) obtained directly from study authors.55–57 Using IPD would increase transparency, improve data quality, offset inadequate reporting within studies, allow for better subgroup analysis, allow for standardizing common measures, and avoid inclusion of studies with a high risk of bias. In-depth assessment of “other” meta-analyses indicates that many were not transparent, had inadequate inclusion/exclusion criteria, and lacked proper risk of bias assessments, across study quality assessments and sensitivity analyses. The same failures occurred in many of these meta-analyses. The combination of including flawed studies and using inadequate meta-analysis techniques rendered many of their conclusions unreliable and useless for clinical decision making. Unfortunately, inadequate and improperly conducted studies of this drug have added to confusion both inside and outside of the medical community. In an August 26, 2021 health advisory, the Centers for Disease Control and Prevention reported a increase in ivermectin the indicating that a substantial number of are this drug for In July the Centers for Disease Control and Prevention reported a increase in to control related to ivermectin with for adverse A more from with patients and and 4 requiring admission after ivermectin of patients had only 3 had received and had an for their These that its absence from COVID-19 treatment guidelines and of and of ivermectin by and has the It is possible that ivermectin studies and meta-analyses to this studies to be by double-blind, placebo-controlled with meta-analyses, using on these studies. an overall COVID-19 of a study with patients each would be to have to detect a or relative in mortality from any (eg, a sicker population Bryant a study with patients each study would be to have to detect a 3.5% in mortality. We results of such studies concluding that treatment with ivermectin can mortality in all or a of patients with COVID-19.

Topics & Concepts