- Hundreds of treatment candidates have been considered for coronavirus SARS-CoV-2.
- As of June 8, 2020, there were 674 trials registered in the US alone.
- Despite the remarkable progress, many trials are flawed by methodological limitations.
- Lack of patient important outcomes, small numbers of patients, and lack of blinding are major limitations in the growing body of evidence for COVID-19 treatments.
- Evidence quality for hydroxychloroquine and remdesivir is low, although these drugs have been widely popularized in the media.
- Skepticism during this period of rapid treatment recommendations remains high until higher
quality studies are available.
“The adequately powered, comparative, and robust clinical research that is needed for optimal evidence-informed decision making remains absent in COVID-19.” – Alexander et al 2020 (1)
COVID-19 Treatment at Pandemic Speed
SARS-CoV-2 has rapidly spread globally with over 18 million cases and counting while crippling the healthcare systems and economies of many countries. Given the magnitude of the detrimental impact of the COVID-19 pandemic, as well as its uncertain future discourse, the scientific community has rushed to identify safe and effective treatments to combat the global public health crisis (2). In many ways, the progress has been remarkable – scientists have managed to conduct research that typically takes a few years in a few months. However, driven by the urgent need for rapid information, the methodological rigour of these research studies is often compromised. Low-quality evidence leads to weak recommendations for patient care. Examining limitations of the current body of evidence can help identify areas of improvement for future studies, as well as make well informed decisions based on the findings of current studies.
“Current approaches are akin to a “Hail Mary” pass in American football to hope that drugs that have worked against a different virus (e.g. hepatitis C or Ebola) will also work against COVID-19” — Jerry Parks and Jeremy Smith 2020 (3)
The “Kitchen Sink” Approach: Current Treatments for COVID-19
A wide variety of treatments are currently being considered for COVID-19. Many new molecular entities are now under investigation. There are also many therapies that previously received
regulatory approval for treating other diseases, such as severe acute respiratory syndrome (SARS), that are also being evaluated to treat COVID-19 (2). Additionally, there is growing interest in using convalescent plasma as a potential therapy option. There has been a rapid increase in the number of clinical trials within a short period of time. As of March 26, 2020, there were 201 trials globally assessing treatment candidates for COVID-19 (2). Exhibit 1 shows distribution of these trials across different countries, with the largest number of trials being conducted in China.
When the pandemic progressed through the summer, the number of trials continued to increase. As of June 8, 2020, there were 674 trials registered in the US alone—representing a 9-fold increase over a 9-week period (4). Exhibit 2 shows a summary of interventions that are currently being tested in the US, with chloroquines being the most tested intervention.
Despite the rising momentum to capitalize on the latest scientific understanding and innovation to deliver a treatment for COVID-19, a critical appraisal of the current body of evidence highlights many methodological limitations. This emphasizes the need to practice a precautionary approach when considering treatments for COVID-19.
“While accelerating morbidity and mortality from the COVID19 pandemic has been paralleled by early and rapid clinical investigation, many trials lack features to optimise their scientific value.” — Mehta et al 2020 (2)
Too Many Trials, Too Little High-Quality Evidence
A wide range of methodological limitations have been identified in the trials assessing treatment candidates for COVID-19. Many of these trials exclude clinical endpoints, have been designed to enroll less than 100 patients, are open label, and use diverse outcome measures (2,5). Preprints of studies have been gaining popularity to give early access to results. However, many preprints are poorly reported, which includes lack of adequate details on the sample frame as well as lack of documentation of missing cases (5). Given this context, studies thus far are likely to provide preliminary results for the safety and effectiveness of treatment candidates for COVID-19 at best – if not add more noise than signal to the current pool of evidence (2,5).
In recent months, several antiviral drugs have received much attention in the media as potential treatment options for COVID-19, without high-quality evidence to support their use. As an example, the case of chloroquine and hydroxychloroquine demonstrates the low methodological quality of current COVID-19 research. When randomized and non-randomized studies of these drugs (published between January 2019 to April 3, 2020) were critically appraised using appropriate risk of bias tools, they all showed high risk of biased estimates of effect (1). In terms of study design, most of the randomized controlled trials (RCTs) did not have randomization, concealment of the generated sequence, as well as the blinding needed to yield sound evidence (1). The sample size of the randomized studies ranged from 22 to 62, whereas the sample size for non-randomized studies ranged from 11 to 80 (6-11). In terms of analysis, the studies did not implement steps necessary to minimize confounding such as statistical adjustment of prognostic factors, propensity matching, or stratification (1).
Beyond the evidence for chloroquines, the current major limitation of COVID-19 research is the lack of large RCTs that can balance prognostic factors with rigorous study design (1). In addition to poor reporting in studies, patient important outcomes that are needed to inform decisions are either not studied or not reported (1). Importantly, the evidence for many treatment candidates currently considered for COVID-19 is indirect – as they come from studies on SARS, influenza, and Middle East respiratory syndrome – and have been rated as low-quality evidence given methodological concerns (12).
Low-quality body of evidence for COVID-19 treatment candidates has nonetheless led to inconsistent recommendations by major governing bodies. While these recommendations are not entirely
supported by evidence, they illuminate the complex factors that come in play when making urgent decisions to protect public health during times of crises.
“You need to be skeptical about [COVID-19] treatments…if you really want to know, wait for the randomized trials and you are in luck.”
“Different values and preferences could lead to very different recommendations based on the given data.” — Dr. Gordon Guyatt, OE World Tour Session: Treating COVID-19 Infection: Does Anything Work?
Remdesivir: It’s About Values and Preferences
Values and preferences of different stakeholders play an important role in recommending different treatment options. During an OE World Tour Session, Dr. Gordon Guyatt illustrated this point by using remdesivir as an example, which has been recently recommended to treat COVID-19. He explained that emerging evidence shows this antiviral drug can lower risk of mortality by 34% –
however, the estimate is not precise (confidence intervals span from a 60% relative risk reduction to a 14% relative risk increase). The quality of evidence available for remdesivir is graded as low-quality, yet this drug received media attention because it lowers the duration of symptoms by three days. However, the confidence interval includes a reduction in the duration of symptoms of less than a day, which is not considered to be a patient important effect, thus evidence is rated down for imprecision. There are now debates about whether evidence for this drug should be rated down to low-quality, as some patients may consider the reduction in the duration of symptoms of less than a day to be worthwhile. Notably, the National Institutes of Health conducted the remdesivir trial and made strong recommendations in favor of this drug – they feel everyone presented with the data will choose this drug. Dr. Guyatt conducted a poll during the OE World Tour Session where he asked attendees whether they would recommend remdesivir to patients if high-quality evidence was available; only some attendees indicated they would recommend this drug, which demonstrates varying preferences. This illustrates that different values and preferences could lead to vastly different recommendations given the same data. During a pandemic, values and preferences can play a particularly important role when governing bodies are faced with the challenge of making swift decisions with only low-quality evidence available.
“High-quality evidence generated by appropriately powered and controlled trials is needed to advance care for patients with coronavirus disease…and those who are susceptible to it.” — Kouzy et al 2020 (4)
A Move Towards High-Quality Evidence for COVID-19 Treatments
Given the magnitude of the current global health crisis, the value of expedited research can not be denied. While the challenges of conducting rigorous research during a time when health care
settings are overwhelmed by the virus outbreak is acknowledged, the emergency of the current situation does not justify transforming flawed methods and data into credible results (1). The
pandemic is here to stay for the foreseeable future and methodological rigour in research studies is needed to generate trustworthy evidence to support optimal decision making (1).
The road to high-quality evidence for COVID-19 treatment candidates can begin with not compromising on the critical design elements when conducting clinical trials – even if it means spending a little more time to get them off the ground. Since RCTs remain the gold standard for determining the safety and efficacy for new therapies, conducting methodologically robust studies that follow this design is going to generate the largest return on investment to provide necessary information for treatment candidates. Scientists should ensure RCTs collect data on relevant covariates, meet sample size requirements for proposed hypotheses and analysis plans, as well as minimize the different sources of bias. The value of using a flexible study framework, as seen in adaptive trials, is increasingly recognized given the need for rapid and efficient study designs during a pandemic. Adaptive trials facilitate the addition of and switch to different treatments as soon as the ones under study are proven to be ineffective or more promising alternatives are available (1). Recently, the World Health Organization (WHO) published a master protocol for randomized multicenter adaptive clinical trials to evaluate the safety and efficacy of new therapies for COVID-19 in combination with standard of care, which can be helpful for scientists to refine and improve the methodologies for their own RCTs (13). In addition to ensuring methodological rigour of research, reporting of studies need to adhere to published guidelines such as the CONSORT (Consolidated
Standards of Reporting Trials) checklist (14). Transparent and comprehensive reporting is particularly important in the context of a pandemic given the potentially large detrimental impact of decisions based on low-quality evidence. Exhibit 3 shows key aspects of meaningful research enterprises during a crisis.
“Use of medication without established effectiveness can undermine public trust, result in unnecessary harm, compromise investigations that might provide definitive answers and divert resources from truly beneficial interventions.” — Kouzy et al 2020 (4)
Navigating the Future
The challenge ahead is not trivial. The virus continues to spread – with multiple countries experiencing peaks and waves as they try to balance reopening their economies with community health. Whether a vaccine will be effective remains unknown. Whether compliance with vaccination is high remains unknown. Whether mutations in the virus limit the effectiveness of a vaccination program remains unknown. It may just be that a treatment for the virus may serve as a critical factor in returning to normality. During this rush to find a treatment, there have been exceedingly optimistic assessments of several treatment candidates, with overestimation of benefits and underestimation of potential harms (12). Even though these hopeful assessments were made by experts and regulatory authorities, the need to critically appraise evidence as a clinician is now more important than ever. The future discourse of this pandemic is uncertain and so is the availability of a safe and effective treatment with high-quality evidence. Despite the external forces the medical community may be subject to in the coming months, making clinical care decisions based on an unbiased assessment of the available evidence will be critical for protecting the health of patients.
OE Community Perspectives on COVID-19 Treatments
We conduced a poll within the OE community to gain their perspectives on COVID-19 treatments. Overall, 42% of participants voted that reduction in the risk of mortality is the most important
outcome to consider when adopting a COVID-19 treatment. Additionally, 60% of the participants indicated that a very large clinical trial that shows that a COVID-19 treatment candidate works will be necessary for them to accept that treatment. These findings reiterate the importance of considering patient important outcomes and large clinical trials for selecting treatments for COVID-19.
Mohit Bhandari, MD, PhD
Dr. Mohit Bhandari is a Professor of Surgery and University Scholar at McMaster University, Canada. He holds a Canada Research Chair in Evidence-Based Orthopaedic Surgery and serves as the Editor-in-Chief of OrthoEvidence.
Ayesha Siddiqua MSc, PhD
Ayesha Siddiqua has a Masters and a PhD from the Health Research Methodology Program in the Department of Health Research Methods, Evidence, and Impact at McMaster University.
1.Alexander PE, Debono VB, Mammen MJ, Iorio A, Aryal K, Deng D, et al (2020). COVID-19 coronavirus research has overall low methodological quality thus far: Case in point for chloroquine/hydroxychloroquine. Journal of Clinical Epidemiology; 123: 120-126. DOI: 10.1016/j.jclinepi.2020.04.016
2. Mehta HB, Ehrhardt S, Moore TJ, Segel JB, & Alexander GC (2020). Characteristics of registered clinical trials assessing treatments for COVID-19: A cross-sectional analysis. BMJ Open; 10: e039978. DOI: 10.1136/bmjopen-2020-039978
3. Parks JM & Smith JC (2020). How to discover antiviral drugs quickly. New England Journal of Medicine; 382(23): 2261-2264. DOI: 10.1056/NEJMcibr2007042
4. Kouzy R, Jaoude JA, Garcia CJG, El Alam MB, Taniguchi CM, & Lubmir EB (2020). Characteristics of the multiplicity of randomized clinical trials for coronavirus disease 2019 launched during the pandemic. JAMA Network Open; 3(7): e2015100. DOI: 10.1001/jamanetworkopen.2020.15100
5. Glasziou PP (2020). Waste in covid-19 research. BMJ; 369: m1847. DOI: 10.1136/bmj.m1847
6.Chen J, Liu D, Liu L, Liu P, Xu Q, Xia L, et al (2020). A pilot study of hydroxychloroquine in treatment of patients with common coronavirus disease19 (COVID-19). The Journal of Zhejiang University; 49(2): 215-219. DOI: 10.3785/j.issn.1008-9292.2020.03.03
7. Chen Z, Hu Z, Zhang Z, Jiang S, Han S, Yan D, et al. Efficacy of hydroxychloroquine in patients with COVID-19: results of a randomized clinical trial. medRxiv. Preprint Published Online First. 10 April 2020. DOI: 10.1101/2020.03.22.20040758
8.Huang M, Tang T, Pang P, Li M, Ma R, Lu J, et al (2020). Treating COVID-19 with chloroquine. Journal of Molecular Cell Biology; 12(4): 322–325. DOI: 10.1093/jmcb/mjaa014
9. Gautret P, Lagier JC, Parola P, Hoang VT, Meddeb L, Mailhe M, et al (2020). Hydroxychloroquine and azithromycin as a treatment of COVID-19: Results of an open-label non-randomized clinical trial. International Journal of Antimicrobial Agents; 56(1): 105949. DOI: 10.1016/j.ijantimicag.2020.105949
10. Gautret P, Lagier JC, Parola P, Hoang VT, Meddeb L, Sevestre J, et al. Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: A pilot observational study. Travel Medicine and Infectious Disease. Published Online First. 11 April 2020. DOI: 10.1016/j.tmaid.2020.101663
11. Molina JM, Delaugerre C, Goff JL, Mela-Lima B, Ponscarme D, Goldwirt L, et al (2020). No evidence of Rapid Antiviral Clearance or clinical benefit with the combination of hydroxychloroquine and azithromycin in patients with severe COVID-19 infection. Medecine et Maladies Infectieuses; 50(4):384. DOI: 10.1016/j.medmal.2020.03.006
12.Ye Z, Rochwerg B, Wang Y, Adhikari NK, Murthy S, Lamontagne F, et al (2020). Treatment of patients with nonsevere and severe coronavirus disease 2019: An evidence-based guideline. Canadian Medical Association Journal; 192(20): E536-E545. DOI: 10.1503/cmaj.200648
13. World Health Organization (2020, February 18). WHO R&D blueprint novel coronavirus COVID-19 therapeutic trial synopsis. Retrieved from https://www.who.int/blueprint/priority-diseases/key-action/COVID-19_Treatment_Trial_Design_Master_Protocol_synopsis_Final_18022020.pdf
14. Schulz KF, Chalmers I, Hayes RJ, & Altman DG (1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273(5): 408-412. DOI: 10.1001/jama.1995.03520290060030