As more outside experts have had a chance to review the huge observational study released last week on the safety and efficacy of hydroxychloroquine (HCQ) and chloroquine for COVID-19, whispers that something was amiss have turned into a loud buzz.
The analysis, published on Friday in The Lancet, looked at nearly 100,000 COVID-19 patients including about 15,000 treated with the antimalarials, either with or without an antibiotic. HCQ was associated with nearly doubled risk of death in the hospital and about 20-fold higher rates of ventricular arrhythmias, the investigators reported.
But other researchers looking at the fine print had questions.
"The claim to have captured data from over 60,000 hospitalizations at over 550 hospitals in North America by April 13th concerns me, given that there were approximately 60,000 COVID-19 hospitalizations total from approximately 6,000 hospitals across all of the United States through April 13th," Matthew Spinelli, MD, of University of California San Francisco, told 鶹ý.
Data from the through April 13 bore him out.
Sapan Desai, MD, PhD, one of the Lancet authors and founder of ., a physician-led public service organization in Chicago that provided much of the data for the analysis, told 鶹ý there were multiple reasons for the discrepancy between the data in the study and that in the COVID Tracking Project.
"There is often a delay before public health reporting catches up to data at the hospital level," he said. Desai also pointed to "issues with data capture at the public health level from various hospitals that could lead to inaccuracies or delays in public reporting."
Walid Gellad, MD, of the University of Pittsburgh, noted on Twitter that according to the Lancet authors, which is "more than the number of deaths in Australia on April 20."
"Not one healthcare facility that contributed data is named or acknowledged. I have never seen that unless someone was using a public[ly] available dataset," David Glidden, PhD, also of University of California San Francisco, told 鶹ý.
Desai added they are reviewing the analysis to ensure there are no issues with the data.
Spinelli also raised questions about the mortality data, saying "prior well-done observational studies did not show such a signal for mortality."
Indeed, a recent New England Journal of Medicine study on hydroxychloroquine did not find the same effect size for mortality.
A blog hosted by statisticians at Columbia University in New York City , including the results being confounded by disease severity, lack of hierarchical modeling, and how the data appeared to be aggregated across continents.
Speaking to the latter point, Desai said the sophistication of data retrieval requires they link directly with electronic health records (EHRs); consequently Surgisphere works exclusively with institutions utilizing "well-established EHRs."
"This requirement allows us to only maintain collaborations with top-tier institutions that are supported by the level of data-integrity and sophistication required for such work," Desai said. "Naturally, this leads to the inclusion of institutions that have a tertiary care level of practice and provide quality healthcare that is relatively homogenous around the world."
Already ripple effects from the study are starting to emerge, with NPR reporting the World Health Organization (WHO) temporarily halting the , which aimed to study , including HCQ.
"I believe that whenever a question arises, it is a responsible action to review the ongoing outcomes as a safety measure in a clinical trial. Their stoppage is temporary based on performing such a review," the study's lead author, Mandeep Mehra, MD, of Brigham and Women's Hospital in Boston, told 鶹ý.
Spinelli said the study merits additional review, ideally including the primary data.
"I am concerned that more desperately needed clinical trials may be stopped as a result of this study," he said.
On June 2, The Lancet issued an for the paper from Mehra's group due to "serious scientific questions" about the data, noting that an independent audit about its validity by authors not affiliated with Surgisphere is ongoing.