A priori the most important cases to consider are the treatment delay-response relationship and the shortest delay to treatment (2+ days in this case). The shortest delay to treatment is significant @94% versus all placebo. (Treatment delay data is in the Supplementary Appendix).

A priori we expect an effective treatment to be most beneficial early, with reducing benefit as treatment is delayed. By simulation, assuming that cases occur randomly according to the observed frequency, the probability that the results follow the observed trend of earlier treatment being better, >10% absolute benefit change between days, and >15% average benefit, is 0.2%. Since we have performed 2 tests, conservative Bonferroni adjustment gives us p = 0.004. Statistical significance here has been confirmed by [1] and [2].

Treatment is relatively late, ~70 to 140 hours after exposure, including the shipping delay. Enrollment was up to 4 days after exposure. The paper does not mention the shipping delay but partial details are provided in the study protocol. They are not clear but indicate no shipping on the weekends and a possible 12pm cutoff for same day dispensing and mailing. Assuming that enrollments were evenly distributed between 6am and 12am each day, we get an average of ~46 hours shipping delay. Wiseman et al. have found the delay may be up to 3.5 days. We have asked for shipping details and will update with more accurate values when available. In any case the treatment delay is quite long and there is no overlap with the more typical delays used such as 0 - 36 hours for oseltamivir. Another source of treatment delay is that the reported exposure may not have been the one that gave the patient COVID-19 - people may have been exposed multiple times before the reported exposure.

Authors initially believed that 3 days since exposure (excluding shipping delay) was the maximum delay of interest, they modified this mid-trial to add an additional day delay. With the original trial specification, they found a 30% reduction in cases, p=0.13. If the trial was not ended early, and if the observed trend continued, 95% significance would have been reached after about 420 patients per group, which is less than the original trial specification of 621 patients per group.

The authors conclude "[treatment] did not prevent illness compatible with COVID-19..", but as above this does not appear to match the data. In the context of their chosen statistics, they could say: "the data suggests a benefit for treatment, but when including the additional delay added mid-study, not analyzing the expected trend for earlier intervention being more effective, and with only 107 cases, we have not yet reached >95% statistical significance."

Authors say that they halted the study due to conditional power analysis, but if additional people have the same or even slightly worse results, >95% statistical significance in their metric will be reached, even when including their added 5+ days case. Further, the data is consistent with the possibility that 0 and 1 day delayed treatment is even more effective.

A note about power: it may seem that with 821 participants the study should have relatively high power. The problem is that only 107 had COVID-19, so the sample size is too small. Since relatively few get COVID-19, the number that need to be treated to prevent a case increases, and looks relatively high compared to other studies. But this is a treatment for preventing death in a global pandemic with a current death toll of , and the treatment being studied is very inexpensive with very good and highly studied safety in controlled conditions.

Only 75% of people reported taking the medication as directed. Actual compliance could be lower. In the OFID podcast, Dr. Boulware notes there were fake submissions with 555 numbers that were removed, there may be more fake submissions that were not identified.

Authors test late post-exposure use, primarily in healthcare volunteers. The primary outcome was having COVID-19 within 14 days. The primary outcome is not the most interesting in terms of the pandemic where the main concern is mortality and morbidity.

Secondary outcomes of hospitalization and death are more relevant. The study has a CFR of 0 and IFR of 0. There was no mortality (or post COVID-19 recovery morbidity) reported. They report 2 hospitalizations but do not provide details.

No serious side effects were seen, even with the dosage used which is higher than typically recommended.

Authors had an objective to intervene before the median incubation period of 5-6 days, but intervention is likely to be more effective very early, as with Oseltamivir for example which must be taken within 2 days (and is likely much more effective earlier). See also the NEJM editorial: "In a small-animal model of SARS-CoV-2, prevention of infection or more severe disease was observed only when the antiviral agent was given before or shortly after exposure."

Research shows the placebo used (folate) may be protective for COVID-19 [3].

Thanks to the authors for their very important, innovative, and interesting study, hard work and dedication. We hope they can revise their conclusions, and we hope they can continue the study, perhaps for use within 24 - 48 hours, and ideally with more fine-grained treatment delay information (hourly).

Additional notes from the NEJM editorial: "This trial has many limitations, acknowledged by the investigators. The trial methods did not allow consistent proof of exposure to SARS-CoV-2 or consistent laboratory confirmation that the symptom complex that was reported represented a SARS-CoV-2 infection. Indeed, the specificity of participant-reported COVID-19 symptoms is low, so it is hard to be certain how many participants in the trial actually had COVID-19. Adherence to the interventions could not be monitored, and participants reported less-than-perfect adherence, more notably in the group receiving [treatment]. In addition, those enrolled in the trial were younger (median age, 40 years) and had fewer coexisting conditions than persons in whom severe COVID-19 is most likely to develop, so enrollment of higher-risk participants might have yielded a different result. The trial design raises questions about the expected prevention benefits of [the treatment]. Studies of postexposure prophylaxis are intended to provide an intervention in the shortest possible time to prevent infection. In a small-animal model of SARS-CoV-2 infection, prevention of infection or more severe disease was observed only when the experimental antiviral agent was given before or shortly after exposure. In the current trial, the long delay between perceived exposure to SARS-CoV-2 and the initiation of [treatment] (≥3 days in most participants) suggests that what was being assessed was prevention of symptoms or progression of COVID-19, rather than prevention of SARS-CoV-2 infection."

Note that author's comments also differ from the published conclusion - for example in the OFID podcast Dr. Boulware has said: "There’s probably two reasons – one is either it just doesn’t work, or the other option is we just didn’t get it to them quick enough. So if you read the tea leaves and look at the subgroup analyses, the people that got enrolled within one or two days of exposure did better than the people that did three or four days later." (8/13: we have removed a comment from Dr. Lewis because it was deleted and Dr. Boulware indicates it was incorrect).

We don't know how many people will get COVID-19 in the future, but based on deaths to date, a treatment which is x% effective could have saved:

17% effective could have saved lives.
30% effective could have saved lives.
49% effective could have saved lives.

Please send us corrections, updates, or comments: