Pandemic Science Out of Control

By Jeanne Lenzer, Shannon Brownlee

On September 14, 1918, in the midst of the worst pandemic in modern history, an article in the New York Times quoted Dr. Rupert Blue, then surgeon general of the US Public Health Service. Blue reported that doctors in many countries were treating their influenza patients with digitalis and the antimalaria drug quinine. There was no evidence that the two drugs were any more effective than folk remedies being used by patients, including cinnamon, goose grease poultices, and salt stuffed up the nose, but doctors were desperate and willing to try just about anything. They would eventually abandon quinine and digitalis as treatments for flu when studies showed they were not only ineffective but caused serious and sometimes deadly side effects.

Today, just shy of two months since the World Health Organization declared COVID-19 a pandemic, the media are once again flooded with cures, patients such as Michigan State Representative Karen Whitsett are being quoted with claims that hydroxychloroquine “saved my life,” and doctors are prescribing drugs that have not been shown to be effective. Only this time, it’s the twenty-first century, the age of “evidence-based medicine.” Or so it might seem. But instead of no science to back up treatments, we now have bad studies being reported uncritically in the press, and Twitter storms of doctors, journalists, and researchers arguing about the ethics of withholding drugs from dying patients, even though we have no idea if those drugs do more harm than good.

If there is a silver lining to all the confusion, it is that this pandemic is exposing three persistent fault lines in medicine. First is the willingness on the part of clinicians to abandon the prime dictum of medicine, to do no harm, and rush into treatments that not only may not work but may well cause serious harm. The fact is, most physicians are not trained to recognize good science from bad. Nor do they have the time to analyze every study, and too many are willing to ignore the need for reliable evidence when fear sets in. Even in non-pandemic times, doctors often favor treatments that have long been in use, seem biologically plausible, are highly remunerative, or have been heavily marketed by manufacturers. In the case of two drugs now being used against COVID-19, hydroxychloroquine and remdesivir, there is a very real possibility that patients who might have recovered from the virus without them will be harmed or even killed by the treatment.

The second issue exposed by the pandemic is the role the media invariably play in hyping science and the physicians who purvey it without regard to the quality of the underlying studies. Some reporters are acting out of ignorance. They know less about good study design than the doctors they quote. Others report on faulty research because it is their business to hype the stock of drug and biotech manufacturers, which have been known to release spurious results of studies when those results favor their products. Whatever the reason, stories in the media help shore up claims of miracle cures made on the basis of poor-quality research.

Third, as has been disturbingly revealed over and over in the past decade or more, too much biomedical research simply isn’t very good. Especially in times of crisis, statistically meaningless, methodologically shoddy, and even fraudulent research can find a receptive audience in desperate patients, credulous doctors, and uninformed journalists.

Medicine without science

To be fair, medicine’s arc toward scientific rigor is not very long. Only in the past few decades have properly performed clinical trials begun to challenge received medical wisdom about the efficacy of common medical interventions such as stents for stable angina, high-dose chemotherapy for breast cancer, vena cava filters to prevent blood clots in the lungs, and ablation for atrial fibrillation, to name just a few. Medicine has a centuries-long tradition of being guided less by careful science and more by personal observation and opinion, especially those of eminent doctors. At times, as with Edward Jenner and the development of smallpox inoculation, such observation has led to great leaps in medical and scientific progress. But perhaps more commonly this tradition has had the effect of propagating useless or harmful medical practices, often for decades and even centuries.

For example, Benjamin Rush, a renowned American physician, philosopher, and statesman, claimed in 1793 that he had saved countless patients during the yellow fever epidemic in Philadelphia by bleeding and purging them. No patient died, he said, if he had bled them at least seven times. The fact that some of his patients didn’t live long enough to be bled seven times apparently was not part of his calculus, and the only true relief his treatment likely offered was to shorten his patients’ suffering by hastening their deaths. Although bloodletting had been popular for millennia, Rush’s claims contributed to its persistence until well into the nineteenth century, when it was finally abandoned after researchers put the practice to a test by comparing two groups: patients subjected to bloodletting, and a control group of those who were not.

Rush was misled into believing in his treatment in part because even the most deadly of epidemics does not kill 100% of its victims. Smallpox patients died only about 30% of the time. Even Ebola has an average case fatality rate of about 50%. A physician can treat quite a few patients who improve and assume that the treatment is what made them recover, while those who die are assumed to have been felled by the disease.

Especially in times of crisis, statistically meaningless, methodologically shoddy, and even fraudulent research can find a receptive audience in desperate patients, credulous doctors, and uninformed journalists.

Today, doctors and researchers are making the same mistake Rush did, and more. Take Didier Raoult, the founder and director of the Institut Hospitalo-Universitaire Méditerranée Infection, in Marseille, France, who was the principal investigator of the study that launched the hubbub over hydroxychloroquine. Raoult and colleagues evaluated 26 patients treated with the drug (six patients were also treated with the antibiotic azithromycin) and 16 control patients who were given supportive care only. Such a small study is inherently problematic. After all, the majority of people with the coronavirus, even those who are hospitalized, will recover. That means a large study is needed in order to trust the results. Jeremy Faust, an emergency physician at Brigham and Women’s Hospital and an instructor at Harvard Medical School, says: “We’re studying a virus in whom 99% of the patients recover. So, if you tell me you did a study of ten or even thirty patients, that’s too small to tell me they wouldn’t have done well anyway.” Thus, the size of Raoult’s study alone should have been a red flag.

More worrisome was the fact that six of the patients Raoult treated with the drug were excluded from the analysis. Why? Because they didn’t complete the full six days of treatment. One died, which makes it difficult to complete any treatment; three got much sicker and had to be transferred to the intensive care unit; and two withdrew from the study. None of the patients in the control arm died, had to be admitted to the ICU, or withdrew.

In other words, all the bad outcomes were among the patients treated with hydroxychloroquine. So how did Raoult manage to claim success? The answer has to do with the use of “surrogate markers,” which are measures, such as laboratory tests, that may or may not have important implications for how well or poorly patients do. In this case, the laboratory measure was a test to detect SARS-CoV-2, the virus that causes COVID-19, in the nose. At day six, 70% of patients who got the drug had cleared the virus from their nose compared with only 12.5% in the control group. In other words, the patients who were treated had less virus, but they still had worse outcomes. Which brings to mind the old joke about the surgeon who comes out of the operating room and announces, “I have good news and bad news. The operation was a success, but the patient died.”

And did we mention that Raoult first publicized the results of this study not in a peer-reviewed medical journal, but on YouTube? Convinced of the efficacy of hydroxychloroquine based on his first, flawed study, Raoult later publicized larger (still unpublished) studies, which he also proclaimed a success. However, in the style of Benjamin Rush, those studies lacked control arms, making it impossible to know what his results mean.

Now add politics

Another study that received attention in the press was conducted by a New York physician, Vladimir (Zev) Zelenko, who claims to have had “100% success” treating 699 patients with a cocktail of hydroxychloroquine, azithromycin, and zinc. His miracle cure is being touted on social media, but Zelenko’s results are even more suspect than Raoult’s. For one thing, most of his patients weren’t even tested for coronavirus, making any claims that he’s curing COVID-19 meaningless. For another, Zelenko’s study was not randomized and had no placebo control arm. Those 699 patients who improved on his drug cocktail may have done no better than patients who got usual care—assuming they even had the disease. Zelenko, like Raoult, also skipped the process of submitting his study for peer review; he took it straight to President Trump’s personal lawyer, former New York City mayor Rudolph Giuliani. Giuliani in turn shared the study with the president and tweeted “Hydroxychloroquine has been shown to have a 100% effective rate treating COVID-19.”

The American Medical Association subsequently joined with two other major health groups in issuing a statement on March 25 urging “extreme caution” for doctors prescribing hydroxychloroquine until data from “robust clinical trials” are available. But that wisdom was countered by the politically conservative American Association of Physicians and Surgeons, which issued a press release on March 30 in support of hydroxychloroquine, stating “prompt or prophylactic treatment with hydroxychloroquine, especially when combined with azithromycin and zinc sulfate, may successfully treat or prevent illness.”

Meanwhile, the US Food and Drug Administration issued an emergency authorization on March 28 giving doctors a green light to prescribe hydroxychloroquine for use against SARS-CoV-2. It’s a mystery how the FDA came to this decision. Certainly the agency has repeatedly come under fire for its cozy relationship with the industry it is supposed to be overseeing. Or perhaps the agency’s commissioner, a political appointee, was taking his cues from the White House after President Trump began promoting hydroxychloroquine in his daily briefings.

Whatever the reason for the FDA’s decision, the result is that 40% of physicians, many of whom have neither the skills nor the time to do their own critical appraisals and need to trust the FDA, say they have prescribed the drug for patients with COVID-19. Some physicians are even hoarding it in case they or family members are infected. Others are denouncing colleagues who urge caution, among them Anthony Fauci, the head of the National Institute of Allergy and Infectious Disease and the most prominent member of the White House Coronavirus Task Force. On an emergency medicine email list, one physician said, “By the time everyone gets their evidence together in a nice controlled study COVID will have packed its bags and left town with thousands dead.” On Twitter, Carole Lieberman, a psychiatrist from Beverly Hills, wrote, “People r dying NOW so no time 4 studies #Fauci wants!”

Now the press and medical communities have turned their attention to remdesivir, an experimental biotech antiviral drug that has never been shown to be effective for any disease. The University of Chicago recruited 125 people for two trials, including 113 with serious symptoms of COVID-19 for a study sponsored by Gilead Pharmaceuticals, the manufacturer of remdesivir. The news outlet STAT obtained a video of researchers at the university reporting on their results to colleagues. STAT quoted Kathleen Mullane, an infectious disease specialist overseeing the studies, as saying, “The best news is that most of our patients have already been discharged, which is great. We’ve only had two patients perish.” The story also quoted Slawomir Michalak, a 57-year-old factory worker who was among the study participants, saying, “Remdesivir was a miracle.” Gilead’s stock shot up when the story broke.

STAT, in its breathless coverage, failed to emphasize that the University of Chicago studies, along with other Gilead-sponsored trials that are running simultaneously, are not randomized and have no placebo control arm. As Mullane even told her colleagues, without a randomized placebo control there is no way to know if the drug works better than no drug at all.

The patient always suffers

The original sin of most of these studies and many others is their lack of a randomized control arm, a key component of clinical trials intended to establish efficacy of any medical test, procedure, device, or drug. Take a recent study from Brazil charting the outcomes of 636 coronavirus outpatients treated with hydroxychloroquine and azithromycin. The researchers say the treatment slashed hospitalization rates from 5.4% to 1.9%. Sounds impressive. It is not. The patients who were in the “control” group weren’t randomized (assigned to the control arm), nor did they receive placebo. Instead they were patients who refused treatment. That means there may be differences in the underlying ages, socioeconomic status, and health of the “control” and treated groups that could easily result in a misleading outcome.

Without a randomized placebo control there is no way to know if a particular drug works better than no drug at all.

Failing to use a properly randomized control arm has repeatedly led to false conclusions in the past. For example, early studies of hormone-replacement therapy (HRT) compared women who chose to take HRT with those who did not. The studies suggested that the therapy cut heart disease in half. But subsequent randomized controlled trials, in which women were assigned to the treatment versus placebo, revealed that the “healthy-user effect” had been at play: women who chose to take HRT (or could afford it) were generally wealthier and healthier than women who didn’t. Randomized controlled trials eventually showed that HRT had no positive impact on heart disease—but it did increase the risk of breast cancer.

As always, it is the patient who suffers the harms of using a drug first and doing the right scientific studies later. In the case of hydroxychloroquine, potential side effects are not trivial. The drug can affect heart rhythm, potentially triggering a rare condition known as torsades de pointes, which can end in sudden cardiac death. One cardiologist posted an EKG tracing of a coronavirus patient on Twitter saying, “Long time since I’ve seen torsades-de-pointe! Careful out there! Please educate your community on the use of #Hydroxycholoroquine.” And a recent study of patients treated with chloroquine (a drug related to hydroxychloroquine and also granted emergency use approval by the FDA) found that 11 of 81 patients died—many with evidence of cardiac injury, which caused the researchers to stop part of the study before it was completed. Unfortunately, that study also did not include a placebo arm on the grounds that the researchers thought it would be “unethical” to deprive study participants access to the drug. But given that there is no idea whether the drug causes more harm than benefit, this appeal to ethics might itself be considered unethical.

A month after telling doctors that prescribing hydroxychloroquine and chloroquine was okay, on April 24 the FDA issued a new advisory suggesting that the drugs should only be prescribed to patients who are in the hospital or enrolled in a clinical trial. While this may or may not cut down on potentially dangerous prescriptions, it won’t do much to improve medical knowledge, because trials enrolling patients in the United States right now lack appropriate control arms, aren’t blinded, are underpowered, or have other serious design flaws.

The lack of randomized control arms in the hydroxychloroquine studies is a problem that the World Health Organization aims to resolve with a trial it calls Solidarity, a multinational, randomized, controlled clinical trial that compares four treatment options against usual care. WHO states that by enrolling patients in one large trial it can facilitate the rapid worldwide comparison of unproven treatments, making it possible to “overcome the risk of multiple small trials not generating the strong evidence needed to determine the relative effectiveness of potential treatments.” But at this point, the cat is out of the bag. By the time those studies are completed, it may be difficult to persuade doctors and patients to stop using drugs they’ve come to believe in even if they turn out to be ineffective or worse.

Fool us twice

Perhaps patients should be forgiven for continuing to grasp at unproven remedies for a frightening disease. But what about the physicians and researchers, not to mention politicians, arguing that there is no time for randomized controlled trials? Many hold up Raoult, Zelenko, the Chicago trial, and other poorly designed studies as proof that hydroxychloroquine and remdesivir work. They are, in effect, saying that it’s better to make a decision based on bad studies now than to wait for the results of properly controlled ones. But given that bad studies are as likely to be wrong as right—and probably more likely to be wrong because they typically are biased toward the result desired by the researcher or study sponsor—why bother with a study at all? Why not try anything—prayer, iridology, homeopathy? At least those are less likely than the drugs to cause dangerous side effects.

The rush to offer unproven treatments outside of well-designed clinical studies undermines high-quality science and condemns us to repeat age-old errors.

Everyone claims to believe in the importance of science, but far too many voices in this debate lack understanding of what it takes to conduct good science, and of how inexact, halting, biased, and poorly executed much medical science has often been. Indeed, over the past 40 years, many of the procedures, tests, drugs, and medical devices that doctors once believed to be valuable have been abandoned as they proved useless or even deadly.

The rush to offer unproven treatments outside of well-designed clinical studies undermines high-quality science and condemns us to repeat age-old errors. It is inevitable that some people will die while properly controlled clinical trials are run, but in the end, far fewer may succumb than if early attempts at treatment are based on studies that fail to meet even the most rudimentary standards of scientific rigor. The first rule of a good clinical trial is to establish an appropriate control group. It was the only thing that persuaded doctors to abandon bloodletting. If we don’t follow that rule now, SARS-CoV-2 or another novel virus will roll around again, and we will still have no effective medicines to treat it.