The Case against Randomized Trials is, Fittingly, Anecdotal

I have a lot of respect for Eric Topol, and am a huge fan of his ongoing work to bring new mobile technology to benefit patients.

The Trial of the Future

However, I am simply baffled by this short video he recently posted on his Medscape blog. In it, he argues against the continued use of randomized controlled trials (RCTs) to provide evidence for or against new drugs.

His argument for this is two anecdotes: one negative, one positive. The negative anecdote is about the recently approved drug for melanoma, Zelboraf:

Well, that's great if one can do [RCTs], but often we're talking about needing thousands, if not tens of thousands, of patients for these types of clinical trials. And things are changing so fast with respect to medicine and, for example, genomically guided interventions that it's going to become increasingly difficult to justify these very large clinical trials. 

For example, there was a drug trial for melanoma and the mutation of BRAF, which is the gene that is found in about 60% of people with malignant melanoma. When that trial was done, there was a placebo control, and there was a big ethical charge asking whether it is justifiable to have a body count. This was a matched drug for the biology underpinning metastatic melanoma, which is essentially a fatal condition within 1 year, and researchers were giving some individuals a placebo.

First and foremost, this is simply factually incorrect on a couple extremely important points.

1. Zelboraf was not approved based on any placebo-controlled trials. The phase 1 and phase 2 trials were both single-arm, open label studies. The only phase 3 trial run before FDA approval used dacarbazine in the comparator arm. In fact, of the 34 trials currently listed for Zelboraf on ClinicalTrials.gov, only one has a placebo control: it’s an adjuvant trial for patients whose melanoma has been completely resected, where no treatment may very well be the best option.
2. The Zelboraf trials are not an example of “needing thousands, if not tens of thousands, of patients” for approval. The phase 3 trial enrolled 675 patients. Even adding the phase 1 and 2 trials doesn’t get us to 1000 patients.

Correcting these details take a lot away from the power of this single drug to be a good example of why we should stop using “the sanctimonious [sic] randomized, placebo-controlled clinical trial”.

The second anecdote is about a novel Alzheimer’s Disease candidate:

A remarkable example of a trial of the future was announced in May. For this trial, the National Institutes of Health is working with [Banner Alzheimer's Institute] in Arizona, the University of Antioquia in Colombia, and Genentech to have a specific mutation studied in a large extended family living in the country of Colombia in South America. There is a family of 8000 individuals who have the so-called Paisa mutation, a presenilin gene mutation, which results in every member of this family developing dementia in their 40s. 

Researchers will be testing a drug that binds amyloid, a monoclonal antibody, in just 300 family members. They're not following these patients out to the point of where they get dementia. Instead, they are using surrogate markers to see whether or not the process of developing Alzheimer's can be blocked using this drug. This is an exciting way in which we can study treatments that can potentially prevent Alzheimer's in a very well-demarcated, very restricted population with a genetic defect, and then branch out to a much broader population of people who are at risk for Alzheimer's. These are the types of trials of the future. 

There are some additional disturbing factual errors here – the extended family numbers about 5,000, not 8,000. And estimates of the prevalence of the mutation within that family appear to vary from about one-third to one-half, so it’s simply wrong to state that “every member of this family” will develop dementia.

However, those errors are relatively minor, and are completely overshadowed by the massive irony that this is a randomized, placebo-controlled trial. Only 100 of the 300 trial participants will receive the active study drug, crenezumab. The other 200 will be on placebo.

And so, the “trial of the future” held up as a way to get us out of using randomized, placebo-controlled trials is actually a randomized, placebo-controlled trial itself. I hope you can understand why I’m completely baffled that Topol thinks this is evidence of anything.

Finally, I have to ask: how is this the trial of the future, anyway? It is a short-term study on a highly-selected patient population with a specific genetic profile, measuring surrogate markers to provide proof of concept for later, larger studies. Is it just me, or does that sound exactly like the early lovastatin trials of the mid-1980’s, which tested cholesterol reduction in a small population of patients with severe heterozygous familial hypercholesterolemia? Back to the Future, indeed.


[Image: time-travelling supercar courtesy of Flickr user JoshBerglund19.]

Why Study Anything When You Already Know Everything?

If you’re a human being, in possession of one working, standard-issue human brain (and, for the remainder of this post, I’m going to assume you are), it is inevitable that you will fall victim to a wide variety of cognitive biases and mistakes.  Many of these biases result in our feeling much more certain about our knowledge of the world than we have any rational grounds for: from the Availability Heuristic, to the Dunning-Kruger Effect, to Confirmation Bias, there is an increasingly-well-documented system of ways in which we (and yes, that even includes you) become overconfident in our own judgment.

Over the years, scientists have developed a number of tools to help us overcome these biases in order to better understand the world.  In the biological sciences, one of our best tools is the randomized controlled trial (RCT).  In fact, randomization helps minimize biases so well that randomized trials have been suggested as a means of developing better governmental policy.

However, RCTs in general require an investment of time and money, and they need to be somewhat narrowly tailored.  As a result, they frequently become the target of people impatient with the process – especially those who perhaps feel themselves exempt from some of the above biases.

A shining example of this impatience-fortified-by-hubris can be

4 out of 5 Hammer Doctors agree:
the world is 98% nail.

found in a recent “Speaking of Medicine” blog post by Dr Trish Greenhalgh, with the mildly chilling title Less Research is Needed.  In it, the author finds a long list of things she feels to be so obvious that additional studies into them would be frivolous.  Among the things the author knows, beyond a doubt, is that patient education does not work, and electronic medical records are inefficient and unhelpful. 

I admit to being slightly in awe of Dr Greenhalgh’s omniscience in these matters. 

In addition to her “we already know the answer to this” argument, she also mixes in a completely different argument, which is more along the lines of “we’ll never know the answer to this”.  Of course, the upshot of that is identical: why bother conducting studies?  For this argument, she cites the example of coronary artery disease: since a large genomic study found only a small association with CAD heritability, Dr Greenhalgh tells us that any studies of different predictive methods is bound to fail and thus not worth the effort (she specifically mentions “genetic, epigenetic, transcriptomic, proteomic, metabolic and intermediate outcome variables” as things she apparently already knows will not add anything to our understanding of CAD). 

As studies grow more global, and as we adapt to massive increases in computer storage and processing ability, I believe we will see an increase in this type of backlash.  And while physicians can generally be relied on to be at the forefront of the demand for more, not less, evidence, it is quite possible that a vocal minority of physicians will adopt this kind of strongly anti-research stance.  Dr Greenhalgh suggests that she is on the side of “thinking” when she opposes studies, but it is difficult to see this as anything more than an attempt to shut down critical inquiry in favor of deference to experts who are presumed to be fully-informed and bias-free. 

It is worthwhile for those of us engaged in trying to understand the world to be aware of these kinds of threats, and to take them seriously.  Dr Greenhalgh writes glowingly of a 10-year moratorium on research – presumably, we will all simply rely on her expertise to answer our important clinical questions.