Magical Thinking in Clinical Trial Enrollment

The many flavors of wish-based patient recruitment.

[Hopefully-obvious disclosure: I work in the field of clinical trial enrollment.]

When I'm discussing and recommending patient recruitment strategies with prospective clients, there is only one serious competitor I'm working against. I do not tailor my presentations in reaction to what other Patient Recruitment Organizations are saying, because they're not usually the thing that causes me the most problems. In almost all cases, when we lose out on a new study opportunity, we have lost to one opponent:

Need patients? Just add water!

Magical thinking.

Magical thinking comes in many forms, but in clinical trial enrollment it traditionally has two dominant flavors:

• We won’t have any problems with enrollment because we have made it a priority within our organization.
  (This translates to: "we want it to happen, therefore it has to happen, therefore it will happen", but it doesn't sound quite as convincing that way, does it?)
• We have selected sites that already have access to a large number of the patients we need.
  (I hear this pretty much 100% of the time. Even from people who understand that every trial is different and that past site performance is simply not a great predictor of future performance.)

A new form of magical thinking burst onto the scene a few years ago: the belief that the Internet will enable us to target and engage exactly the right patients. Specifically, some teams (aided by the, shall we say, less-than-completely-totally-true claims of "expert" vendors) began to believe that the web’s great capacity to narrowly target specific people – through Google search advertising, online patient communities, and general social media activities – would prove more than enough to deliver large numbers of trial participants. And deliver them fast and cheap to boot. Sadly evidence has already started to emerge about the Internet’s failure to be a panacea for slow enrollment. As I and others have pointed out, online recruitment can certainly be cost effective, but cannot be relied on to generate a sizable response. As a sole source, it tends to underdeliver even for small trials.

I think we are now seeing the emergence of the newest flavor of magical thinking: Big Data. Take this quote from recent coverage of the JP Morgan Healthcare Conference:

For instance, Phase II, that ever-vexing rubber-road matchmaker for promising compounds that just might be worthless. Identifying the right patients for the right drug can make or break a Phase II trial, [John] Reynders said, and Big Data can come in handy as investigators distill mountains of imaging results, disease progression readings and genotypic traits to find their target participants. 

The prospect of widespread genetic mapping coupled with the power of Big Data could fundamentally change how biotech does R&D, [Alexis] Borisy said. "Imagine having 1 million cancer patients profiled with data sets available and accessible," he said. "Think how that very large data set might work--imagine its impact on what development looks like. You just look at the database and immediately enroll a trial of ideal patients."

Did you follow the logic of that last sentence? You immediately enroll ideal patients ... and all you had to do was look at a database! Problem solved!

Before you go rushing off to get your company some Big Data, please consider the fact that the overwhelming majority of Phase 2 trials do not have a neat, predefined set of genotypic traits they’re looking to enroll. In fact, narrowly-tailored phase 2 trials (such as recent registration trials of Xalkori and Zelboraf) actually enroll very quickly already, without the need for big databases. The reality for most drugs is exactly the opposite: they enter phase 2 actively looking for signals that will help identify subgroups that benefit from the treatment.

Also, it’s worth pointing out that having a million data points in a database does not mean that you have a million qualified, interested, and nearby patients just waiting to be enrolled in your trial. As recent work in medical record queries bears out, the yield from these databases promises to be low, and there are enormous logistic, regulatory, and personal challenges in identifying, engaging, and consenting the actual human beings represented by the data.

More, even fresher flavors of magical thinking are sure to emerge over time. Our urge to hope that our problems will just be washed away in a wave of cool new technology is just too powerful to resist.

However, when the trial is important, and the costs of delay are high, clinical teams need to set the wishful thinking aside and ask for a thoughtful plan based on hard evidence. Fortunately, that requires no magic bean purchase.

Magic Beans picture courtesy of Flikr user sleepyneko

Clinical trials: recent reading recommendations

My recommended reading list -- highlights from the past week:


Absolute required reading for anyone who designs protocols or is engaged in recruiting patients into clinical trials: Susan Guber writes eloquently about her experiences as a participant in cancer clinical trials.
New York Times Well Blog: The Trials of Cancer Trials

Today's #FDAFridayPhoto features Harvey
Wiley, leader of the famed FDA "Poison Squad".

The popular press in India continues to be disingenuous and exploitative in its coverage of clinical trial deaths in that country. (My previous thoughts on that are here.) Kiran Mazumdar-Shaw, an industry leader, has put together an intelligent and articulate antidote.
The Economic Times: Need a rational view on clinical trials


Rahlen Gossen exhibits mastery of the understatement: “Though the Facebook Insights dashboard is a great place to start, it has a few significant disadvantages.” She also provides a good overview of the most common pitfalls you’ll encounter when you try to get good metrics out of your Facebook campaign. 

Rebar Interactive Blog: Metrics Gone Wrong: Facebook “Likes” in Clinical Research

I have not had a chance to watch it yet, but I’m excited to see that theHeart.org has just posted a 7-part video editorial series by Yale’s Harlan Krumholz and Duke Stanford’s Bob Harrington on “a frank discussion on the controversies in the world of clinical trials”. 

theHeart.org: Rethinking clinical trials and research in cardiology

Pfizer Shocker: Patient Recruitment is Hard

In what appears to be, oddly enough, an exclusive announcement to Pharmalot, Pfizer will be discontinuing its much-discussed “Trial in a box”—a clinical study run entirely from a patient’s home. Study drug and other supplies would be shipped directly to each patient, with consent, communication, and data collection happening entirely via the internet.

The trial piloted a number of innovations, including some novel and intriguing Patient Reported Outcome (PRO) tools.  Unfortunately, most of these will likely not have been given the benefit of a full test, as the trial was killed due to low patient enrollment.

The fact that a trial designed to enroll less than 300 patients couldn’t meet its enrollment goal is sobering enough, but in this case the pain is even greater due to the fact that the study was not limited to site databases and/or catchment areas.  In theory, anyone with overactive bladder in the entire United States was a potential participant. 

And yet, it didn’t work.  In a previous interview with Pharmalot, Pfizer’s Craig Lipset mentions a number of recruitment channels – he specifically cites Facebook, Google, Patients Like Me, and Inspire, along with other unspecified “online outreach” – that drove “thousands” of impressions and “many” registrations, but these did not amount to, apparently, even close to the required number of consented patients. 

Two major questions come to mind:

1.    How were patients “converted” into the study?  One of the more challenging aspects of patient recruitment is often getting research sites engaged in the process.  Many – perhaps most – patients are understandably on the fence about being in a trial, and the investigator and study coordinator play the single most critical role in helping each patient make their decision. You cannot simply replace their skill and experience with a website (or “multi-media informed consent module”). 

2.    Did they understand the patient funnel?  I am puzzled by the mention of “thousands of hits” to the website.  That may seem like a lot, if you’re not used to engaging patients online, but it’s actually not necessarily so. 

Jakob Nielsen's famous "Lurker Funnel"
seems worth mentioning here...

Despite some of the claims made by patient communities, it is perfectly reasonable to expect that less than 1% of visitors (even somewhat pre-qualified visitors) will end up consenting into the study.  If you’re going to rely on the internet as your sole means of recruitment, you should plan on needing closer to 100,000 visitors (and, critically: negotiate your spending accordingly). 

In the prior interview, Lipset says:

I think some of the staunch advocates for using online and social media for recruitment are still reticent to claim silver bullet status and not use conventional channels in parallel. Even the most aggressive and bullish social media advocates, generally, still acknowledge you’re going to do this in addition to, and not instead of more conventional channels.

This makes Pfizer’s exclusive reliance on these channels all the more puzzling.  If no one is advocating disintermediating the sites and using only social media, then why was this the strategy?

I am confident that someone will try again with this type of trial in the near future.  Hopefully, the Pfizer experience will spur them to invest in building a more rigorous recruitment strategy before they start.

[Update 6/20: Lipset weighed in via the comments section of the Pharmalot article above to clarify that other DTP aspects of the trial were tested and "worked VERY well".  I am not sure how to evaluate that clarification, given the fact that those aspects couldn't have been tested on a very large number of patients, but it is encouraging to hear that more positive experiences may have come out of the study.]

Social Networking for Clinical Research

No matter what, negative clinical trial results are sad. We can appreciate, intellectually, that clinical equipoise is important and that negative results are a natural consequence of conducting ethical trials, but it is impossible not to feel disappointed when yet another promising therapy fails to hold up.

However, the negative results published today in Nature Biotechnology on a groundbreaking trial in ALS deserve to be celebrated. The trial was conducted exclusively through PatientsLikeMe, the online medical social network that serves as a forum for patients in all disease areas to “share real-world health experiences.”

According to a very nice write-up in the Wall Street Journal, the trial was conceived and initiated by ALS patients who were part of the PatientsLikeMe ALS site:

Jamie Heywood, chairman and co-founder of PatientsLikeMe, said the idea for the new study came from patients. After the 2008 paper reporting lithium slowed down the disease in 16 ALS patients, some members of the site suggested posting their experiences with the drug in an online spreadsheet to figure out if it was working. PatientsLikeMe offered instead to run a more rigorous observational study with members of the network to increase chances of getting a valid result.

The study included standardized symptom reporting from 596 patients (149 taking lithium and 447 matched controls). After 9 months, the patients taking lithium showed almost no difference in ALS symptoms compared to their controls, and preliminary (negative) results were released in late 2008. Although the trial was not randomized and not blinded – significant methodological issues, to be sure – it is still exciting for a number of reasons.

First, the study was conducted at an incredible rate of speed. Only 9 months elapsed between PatientsLikeMe deploying its tool to users and the release of topline results. In contrast, 2 more traditional, controlled clinical trials that were initiated to verify the first study’s results had not even managed to enroll their first patient during that time. In many cases like this – especially looking at new uses of established, generic drugs – private industry has little incentive to conduct an expensive trial. And academic researchers tend to move a pace that, while not quite glacial, is not as rapid as acutely-suffering patients would like.

(The only concern I have about speed is the time it took to get this paper published. Why was there a 2+ year gap between results and publication?)

Second, this trial represents one of the best uses of “off-label” patient experience that I know of. Many of the physicians I talk to struggle with off-label, patient-initiated treatment: they cannot support it, but it is difficult to argue with a patient when there is so little hard evidence. This trial represents an intelligent path towards tapping into and systematically organizing some of the thousands of individual off-label experiences and producing something clinically useful. As the authors state in the Nature paper:

Positive results from phase 1 and phase 2 trials can lead to changes in patient behavior, particularly when a drug is readily available. [...] The ongoing availability of a surveillance mechanism such as ours might help provide evidence to support or refute self-experimentation.

Ironically, the fact that the trial found no benefit for lithium may have the most far-reaching benefit. A positive trial would have been open to criticism for its inability to compensate for placebo effect. These results run counter to expected placebo effect, lending strong support to the conclusion that it was thoughtfully designed and conducted. I hope this will be immense encouragement to others looking to take this method forward.

A lot has been written over the past 3-4 years about the enormous power of social media to change healthcare as we know it. In general, I have been skeptical of most of these claims, as most of them fail to plausibly explain the connection between "Lots of people on Facebook" and "Improved clinical outcomes". I applaud the patients and staff at PatientsLikeMe for finding a way to work together to break new ground in this area.