Patient Recruitment: Taking the Low Road

The Wall Street Journal has an interesting article on the use of “Big Data” to identify and solicit potential clinical trial participants. The premise is that large consumer data aggregators like Experian can target patients with certain diseases through correlations with non-health behavior. Examples given include “a preference for jazz” being associated with arthritis and “shopping online for clothes” being an indicator of obesity.

We've seen this story before.

In this way, allegedly, clinical trial patient recruitment companies can more narrowly target their solicitations* for patients to enroll in clinical trials.

In the spirit of full disclosure, I should mention that I was interviewed by the reporter of this article, although I am not quoted. My comments generally ran along three lines, none of which really fit in with the main storyline of the article:

1. I am highly skeptical that these analyses are actually effective at locating patients
   
2. These methods aren't really new – they’re the same tactics that direct marketers have been using for years
   
3. Most importantly, the clinical trials community can – and should – be moving towards open and collaborative patient engagement. Relying on tactics like consumer data snooping and telemarketing is an enormous step backwards.

The first point is this: certainly some diseases have correlates in the real world, but these correlates tend to be pretty weak, and are therefore unreliable predictors of disease. Maybe it’s true that those struggling with obesity tend to buy more clothes online (I don’t know if it’s true or not – honestly it sounds a bit more like an association built on easy stereotypes than on hard data). But many obese people will not shop online (they will want to be sure the clothes actually fit), and vast numbers of people with low or average BMIs will shop for clothes online.  So the consumer data will tend to have very low predictive value. The claims that liking jazz and owning cats are predictive of having arthritis are even more tenuous. These correlates are going to be several times weaker than basic demographic information like age and gender. And for more complex conditions, these associations fall apart.

Marketers claim to solve this by factoring a complex web of associations through a magical black box – th WSJ article mentions that they “applied a computed algorithm” to flag patients. Having seen behind the curtain on a few of these magic algorithms, I can confidently say that they are underwhelming in their sophistication. Hand-wavy references to Big Data and Algorithms are just the tools used to impress pharma clients. (The down side to that, of course, is that you can’t help but come across as big brotherish – see this coverage from Forbes for a taste of what happens when people accept these claims uncritically.)

But the effectiveness of these data slice-n-dicing activities is perhaps beside the point. They are really just a thin cover for old-fashioned boiler room tactics: direct mail and telemarketing. When I got my first introduction to direct marketing in the 90’s, it was the exact same program – get lead lists from big companies like Experian, then aggressively mail and call until you get a response.

The limited effectiveness and old-school aggressiveness of these programs comes is nicely illustrated in the article by one person’s experience:

Larna Godsey, of Wichita, Kan., says she received a dozen phone calls about a diabetes drug study over the past year from a company that didn't identify itself. Ms. Godsey, 63, doesn't suffer from the disease, but she has researched it on the Internet and donated to diabetes-related causes. "I don't know if it's just a coincidence or if they're somehow getting my information," says Ms. Godsey, who filed a complaint with the FTC this year.

The article notes that one recruitment company, Acurian, has been the subject of over 500 FTC complaints regarding its tactics. It’s clear that Big Data is just the latest buzzword lipstick on the telemarketing pig. And that’s the real shame of it.

We have arrived at an unprecedented opportunity for patients, researchers, and private industry to come together and discuss, as equals, research priorities and goals. Online patient communities like Inspire and PatientsLikeMe have created new mechanisms to share clinical trial opportunities and even create new studies. Dedicated disease advocates have jumped right into the world of clinical research, with groups like the Cystic Fibrosis Foundation and Michael J. Fox Foundation no longer content with raising research funds, but actively leading the design and operations of new studies.

Some – not yet enough – pharmaceutical companies have embraced the opportunity to work more openly and honestly with patient groups. The scandal of stories like this is not the Wizard of Oz histrionics of secret computer algorithms, but that we as an industry continue to take the low road and resort to questionable boiler room tactics.

It’s past time for the entire patient recruitment industry to drop the sleaze and move into the 21st century. I would hope that patient groups and researchers will come together as well to vigorously oppose these kinds of tactics when they encounter them.

(*According to the article, Acurian "has said that calls related to medical studies aren't advertisements as defined by law," so we can agree to call them "solicitations".)

Half of All Trials Unpublished*

(*For certain possibly nonstandard uses of the word "unpublished")

This is an odd little study. Instead of looking at registered trials and following them through to publication, this study starts with a random sample of phase 3 and 4 drug trials that already had results posted on ClinicalTrials.gov - so in one, very obvious sense, none of the trials in this study went unpublished.

Timing and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals
Carolina Riveros, Agnes Dechartres, Elodie Perrodeau, Romana Haneef, Isabelle Boutron, Philippe Ravaud

But here the authors are concerned with publication in medical journals, and they were only able to locate journal articles covering about half (297/594) of trials with registered results. 

It's hard to know what to make of these results, exactly. Some of the "missing" trials may be published in the future (a possibility the authors acknowledge), some may have been rejected by one or more journals (FDAAA requires posting the results to ClinicalTrials.gov, but it certainly doesn't require journals to accept trial reports), and some may be pre-FDAAA trials that sponsors have retroactively added to ClinicalTrials.gov even though development on the drug has ceased.

It would have been helpful had the authors reported journal publication rates stratified by the year the trials completed - this would have at least given us some hints regarding the above. More than anything I still find it absolutely bizarre that in a study this small, the entire dataset is not published for review.

One potential concern is the search methodology used by the authors to match posted and published trials. If the easy routes (link to article already provided in ClinicalTrials.gov, or NCT number found in a PubMed search) failed, a manual search was performed:

The articles identified through the search had to match the corresponding trial in terms of the information registered at ClinicalTrials.gov (i.e., same objective, same sample size, same primary outcome, same location, same responsible party, same trial phase, and same sponsor) and had to present results for the primary outcome. 

So it appears that a reviewed had to score the journal article as an exact match on 8 criteria in order for the trial to be considered the same. That could easily lead to exclusion of journal articles on the basis of very insubstantial differences. The authors provide no detail on this; and again, that would be easy to verify if the study dataset was published. 

The reason I harp on this, and worry about the matching methodology, is that two of the authors of this study were also involved in a methodologically opaque and flawed study about clinical trial results posted in the JCO. In that study, as well, the authors appeared to use an incorrect methodology to identify published clinical trials. When I pointed the issues out, the corresponding author merely reiterated what was already (insufficiently) in the paper's Methodology section.

I find it strange beyond belief, and more than a little hypocritical, that researchers would use a public, taxpayer-funded database as the basis of their studies, and yet refuse to provide their data for public review. There are no technological or logistical issues preventing this kind of sharing, and there is an obvious ethical point in favor of transparency.

But if the authors are reasonably close to correct in their results, I'm not sure what to make of this study. 

The Nature article covering this study contend that

[T]he [ClinicalTrials.gov] database was never meant to replace journal publications, which often contain longer descriptions of methods and results and are the basis for big reviews of research on a given drug.

I suppose that some journal articles have better methodology sections, although this is far from universally true (and, like this study here, these methods are often quite opaquely described and don't support replication). As for results, I don't believe that's the case. In this study, the opposite was true: ClinicalTrial.gov results were generally more complete than journal results. And I have no idea why the registry wouldn't surpass journals as a more reliable and complete source of information for "big reviews".

Perhaps it is a function of my love of getting my hands dirty digging into the data, but if we are witnessing a turning point where journal articles take a distant back seat to the ClinicalTrials.gov registry, I'm enthused. ClinicalTrials.gov is public, free, and contains structured data; journal articles are expensive, unparsable, and generally written in painfully unclear language. To me, there's really no contest. 

Carolina Riveros, Agnes Dechartres, Elodie Perrodeau, Romana Haneef, Isabelle Boutron, & Philippe Ravaud (2013). Timing and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals PLoS Medicine DOI: 10.1371/journal.pmed.1001566

Brave New Biopharm Blogging

Although a few articles on this site are older, I really only began blogging in earnest about 15 months ago. However, I suppose that's long enough that I can count myself as at least somewhat established, and take a moment to welcome and encourage some interesting newcomers to the scene.

 

Bloggers in dank basements their natural habitat.

There are 3 relative newcomers that I've found really interesting, all with very different perspectives on drug development and clinical research:

Pharmagellan 

The Big Pharma insider.

With the exception of John LaMattina (the former Pfizer exec who regularly provides seriously thought provoking ideas over on Forbes), I don’t know of anyone from the ranks of Big Pharma who writes both consistently and well. Which is a shame, given how many major past, current, and future therapies pass through those halls.

Enter Frank David, the Director of Strategy at AstraZeneca's Oncology Innovative Medicines unit. Frank started his Pharmagellan blog this April, and has been putting out a couple thoughtful perspective pieces a month since then.

Frank also gets my vote for most under-followed Twitter account in the industry, as he’s putting out a steady stream of interesting material.

ClinOps Toolkit

Getting trials done.

Clinical operations – the actual execution of the clinical trials we all talk about – is seriously underrepresented in the blogosphere. There are a number of industry blogs, but none that aren’t trying first and foremost to sell you something.

I met Nadia Bracken on my last trip out to the San Francisco bay area. To say Nadia is driven is to make a rather silly understatement. Nadia is driven. She thinks fast and she talks fast. ClinOps Toolkit is a blog (or resource? or community?) that is still very much in development, but I think it holds a tremendous amount of potential. People working in ClinOps should be embracing her, and those of us who depend on operations teams getting the job done should keep a close eye on the website.

Other Secrets of the Buy Side

Watching the money.

I am not a stock trader. I am a data person, and data says trust big sample sizes. And, honestly, I just don't have the time.

But that doesn't stop me from realizing that a lot of great insight about drug development – especially when it concerns small biotechs – is coming from the investment community. So I tend to follow a number of financial writers, as I've found that they do a much better job of digging through the hype than can ever be expected of the mainstream media.

One stock writer who I've been following for a while is Andrew Goodwin, who maintains the Biotech Due Diligence website and blog. Andrew clearly has a great grasp on a number of topics, so when he described a new blog as a “must-have addition” to one's reading list, I had to take a look.

And the brand-new-this-month blog, by David Sable at Special Situations Fund, does seem like a great read. David looks both at the corporate dynamics and scientific stories of biotechs with a firmly skeptical view. I know most blogs this new will not be around 6 months from now (and David admits as much in his opening post), but I’m hoping this one lasts.

. . . . .

So, I encourage you to take a look at the above 3 blogs. I'm happy to see more and diverse perspectives on the drug development process starting to emerge, and hope that all 3 of these authors stick around for quite a while – we need their ideas.

[Bloggerhole photo courtesy of Flikr user second_mouse.]

Questionable Enrollment Math(s) - the Authors Respond

The authors of the study I blogged about on Monday were kind enough to post a lengthy comment, responding in part to some of the issues I raised. I thought their response was interesting, and so reprint it in its entirety below, interjecting my own reactions as well.

There were a number of points you made in your blog and the title of questionable maths was what caught our eye and so we reply on facts and provide context.

Firstly, this is a UK study where the vast majority of UK clinical trials take place in the NHS. It is about patient involvement in mental health studies - an area where recruitment is difficult because of stigma and discrimination.

I agree, in hindsight, that I should have titled the piece “questionable maths” rather than my Americanized “questionable math”. Otherwise, I think this is fine, although I’m not sure that anything here differs from my post.

1. Tripling of studies - You dispute NIHR figures recorded on a national database and support your claim with a lone anecdote - hardly data that provides confidence. The reason we can improve recruitment is that NIHR has a Clinical Research Network which provides extra staff, within the NHS, to support high quality clinical studies and has improved recruitment success.

To be clear, I did not “dispute” the figures so much as I expressed sincere doubt that those figures correspond with an actual increase in actual patients consenting to participate in actual UK studies. The anecdote explains why I am skeptical – it's a bit like I've been told there was a magnitude 8 earthquake in Chicago, but neither I nor any of my neighbors felt anything. There are many reasons why reported numbers can increase in the absence of an actual increase. It’s worth noting that my lack of confidence in the NIHR's claims appears to be shared by the 2 UK-based experts quoted by Applied Clinical Trials in the article I linked to.

2. Large database: We have the largest database of detailed study information and patient involvement data - I have trawled the world for a bigger one and NIMH say there certainly isn't one in the USA. This means few places where patient impact can actually be measured

3. Number of studies: The database has 374 studies which showed among other results that service user involvement increased over time probably following changes by funders e.g. NIHR requests information in the grant proposal on how service users have been and will be involved - one of the few national funders to take this issue seriously.

As far as I can tell, neither of these points is in dispute.

4. Analysis of patient involvement involves the 124 studies that have completed. You cannot analyse recruitment success unless then.

I agree you cannot analyze recruitment success in studies that have not yet completed. My objection is that in both the KCL press release and the NIHR-authored Guardian article, the only number mentioned in 374, and references to the recruitment success findings came immediately after references to that number. For example:

Published in the British Journal of Psychiatry, the researchers analysed 374 studies registered with the Mental Health Research Network (MHRN).

Studies which included collaboration with service users in designing or running the trial were 1.63 times more likely to recruit to target than studies which only consulted service users.  Studies which involved more partnerships - a higher level of Patient and Public Involvement (PPI) - were 4.12 times more likely to recruit to target.

The above quote clearly implies that the recruitment conclusions were based on an analysis of 374 studies – a sample 3 times larger than the sample actually used. I find this disheartening.

The complexity measure was developed following a Delphi exercise with clinicians, clinical academics and study delivery staff to include variables likely to be barriers to recruitment. It predicts delivery difficulty (meeting recruitment & delivery staff time). But of course you know all that as it was in the paper.

Yes, I did know this, and yes, I know it because it was in the paper. In fact, that’s all I know about this measure, which is what led me to characterize it as “arbitrary and undocumented”. To believe that all aspects of protocol complexity that might negatively affect enrollment have been adequately captured and weighted in a single 17-point scale requires a leap of faith that I am not, at the moment, able to make. The extraordinary claim that all complexity issues have been accounted for in this model requires extraordinary evidence, and “we conducted a Delphi exercise” does not suffice.  

6. All studies funded by NIHR partners were included – we only excluded studies funded without peer review, not won competitively. For the involvement analysis we excluded industry studies because of not being able to contact end users and where inclusion compromised our analysis reliability due to small group sizes.

It’s only that last bit I was concerned about. Specifically, the 11 studies that were excluded due to being in “clinical groups” that were too small, despite the fact that “clinical groups” appear to have been excluded as non-significant from the final model of recruitment success.

(Also: am I being whooshed here? In a discussion of "questionable math" the authors' enumeration goes from 4 to 6. I’m going to take the miscounting here as a sly attempt to see if I’m paying attention...)

I am sure you are aware of the high standing of the journal and its robust peer review. We understand that our results must withstand the scrutiny of other scientists but many of your comments were unwarranted. This is the first in the world to investigate patient involvement impact. No other databases apart from the one held by the NIHR Mental Health Research Network is available to test – we only wish they were.

I hope we can agree that peer review – no matter how "high standing" the journal – is not a shield against concern and criticism. Despite the length of your response, I’m still at a loss as to which of my comments specifically were unwarranted.

In fact, I feel that I noted very clearly that my concerns about the study’s limitations were minuscule compared to my concerns about the extremely inaccurate way that the study has been publicized by the authors, KCL, and the NIHR. Even if I conceded every possible criticism of the study itself, there remains the fact that in public statements, you

1. Misstated an odds ratio of 4 as “4 times more likely to”
2. Overstated the recruitment success findings as being based on a sample 3 times larger than it actually was
3. Re-interpreted, without reservation, a statistical association as a causal relationship
4. Misstated the difference between the patient involvement categories as being a matter of merely “involving just one or two patients in the study team”

And you did these consistently and repeatedly – in Dr Wykes's blog post, in the KCL press release, and in the NIHR-written Guardian article.

To use the analogy from my previous post: if a pharmaceutical company had committed these acts in public statements about a new drug, public criticism would have been loud and swift.

Your comment on the media coverage of odds ratios is an issue that scientists need to overcome (there is even a section in Wikipedia).

It's highly unfair to blame "media coverage" for the use of an odds ratio as if it were a relative risk ratio. In fact, the first instance of "4 times more likely" appears in Dr Wykes's own blog post. It's repeated in the KCL press release, so you yourselves appear to have been the source of the error.

You point out the base rate issue but of course in a logistic regression you also take into account all the other variables that may impinge on the outcome prior to assessing the effects of our key variable patient involvement - as we did – and showed that the odds ratio is 4.12 - So no dispute about that. We have followed up our analysis to produce a statement that the public will understand. Using the following equations:

Model predicted recruitment lowest level of involvement exp(2.489-.193*8.8-1.477)/(1+exp(2.489-.193*8.8-1.477))=0.33

Model predicted recruitment highest level of involvement exp(2.489-.193*8.8-1.477+1.415)/(1+exp(2.489-.193*8.8-1.477+1.415)=0.67

For a study of typical complexity without a follow up increasing involvement from the lowest to the highest levels increased recruitment from 33% to 66% i.e. a doubling.

So then, you agree that your prior use of “4 times more likely” was not true? Would you be willing to concede that in more or less direct English?

This is important and is the first time that impact has been shown for patient involvement on the study success.

Luckily in the UK we have a network that now supports clinicians to be involved and a system for ensuring study feasibility.

The addition of patient involvement is the additional bonus that allows recruitment to increase over time and so cutting down the time for treatments to get to patients.

No, and no again. This study shows an association in a model. The gap between that and a causal relationship is far too vast to gloss over in this manner.

In summary, I thank the authors for taking the time to response, but I feel they've overreacted to my concerns about the study, and seriously underreacted to my more important concerns about their public overhyping of the study. 

I believe this study provides useful, though limited, data about the potential relationship between patient engagement and enrollment success. On the other hand, I believe the public positioning of the study by its authors and their institutions has been exaggerated and distorted in clearly unacceptable ways. I would ask the authors to seriously consider issuing public corrections on the 4 points listed above.

Questionable Enrollment Math at the UK's NIHR

There has been considerable noise coming out of the UK lately about successes in clinical trial enrollment.

First, a couple months ago came the rather dramatic announcement that clinical trial participation in the UK had "tripled over the last 6 years". That announcement, by the chief executive of the

Sweet creature of bombast: is Sir John
writing press releases for the NIHR?

National Institute of Health Research's Clinical Research Network, was quickly and uncritically picked up by the media.

That immediately caught my attention. In large, global trials, most pharmaceutical companies I've worked with can do a reasonable job of predicting accrual levels in a given country. I like to think that if participation rates in any given country had jumped that heavily, I’d have heard something.

(To give an example: looking at a quite-typical study I worked on a few years ago: UK sites were overall slightly below the global average. The highest-enrolling countries were about 2.5 times as fast. So, a 3-fold increase in accruals would have catapulted the UK from below average to the fastest-enrolling country in the world.)

Further inquiry, however, failed to turn up any evidence that the reported tripling actually corresponded to more human beings enrolled in clinical trials. Instead, there is some reason to believe that all we witnessed was increased reporting of trial participation numbers.

Now we have a new source of wonder, and a new giant multiplier coming out of the UK. As the Director of the NIHR's Mental Health Research Network, Til Wykes, put it in her blog coverage of her own paper:

Our research on the largest database of UK mental health studies shows that involving just one or two patients in the study team means studies are 4 times more likely to recruit successfully.

Again, amazing! And not just a tripling – a quadrupling!

Understand: I spend a lot of my time trying to convince study teams to take a more patient-focused approach to clinical trial design and execution. I desperately want to believe this study, and I would love having hard evidence to bring to my clients.

At first glance, the data set seems robust. From the King's College press release:

Published in the British Journal of Psychiatry, the researchers analysed 374 studies registered with the Mental Health Research Network (MHRN).

Studies which included collaboration with service users in designing or running the trial were 1.63 times more likely to recruit to target than studies which only consulted service users.  Studies which involved more partnerships - a higher level of Patient and Public Involvement (PPI) - were 4.12 times more likely to recruit to target.

But here the first crack appears. It's clear from the paper that the analysis of recruitment success was not based on 374 studies, but rather a much smaller subset of 124 studies. That's not mentioned in either of the above-linked articles.

And at this point, we have to stop, set aside our enthusiasm, and read the full paper. And at this point, critical doubts begin to spring up, pretty much everywhere.

First and foremost: I don’t know any nice way to say this, but the "4 times more likely" line is, quite clearly, a fiction. What is reported in the paper is a 4.12 odds ratio between "low involvement" studies and "high involvement" studies (more on those terms in just a bit).  Odds ratios are often used in reporting differences between groups, but they are unequivocally not the same as "times more likely than".

This is not a technical statistical quibble. The authors unfortunately don’t provide the actual success rates for different kinds of studies, but here is a quick example that, given other data they present, is probably reasonably close:

• A Studies: 16 successful out of 20 
• Probability of success: 80% 
• Odds of success: 4 to 1
• B Studies: 40 successful out of 80
• Probability of success: 50%
• Odds of success: 1 to 1

From the above, it’s reasonable to conclude that A studies are 60% more likely to be successful than B studies (the A studies are 1.6 times as likely to succeed). However, the odds ratio is 4.0, similar to the difference in the paper. It makes no sense to say that A studies are 4 times more likely to succeed than B studies.

This is elementary stuff. I’m confident that everyone involved in the conduct and analysis of the MHRN paper knows this already. So why would Dr Wykes write this? I don’t know; it's baffling. Maybe someone with more knowledge of the politics of British medicine can enlighten me.

If a pharmaceutical company had promoted a drug with this math, the warning letters and fines would be flying in the door fast. And rightly so. But if a government leader says it, it just gets recycled verbatim.

The other part of Dr Wykes's statement is almost equally confusing. She claims that the enrollment benefit occurs when "involving just one or two patients in the study team". However, involving one or two patients would seem to correspond to either the lowest ("patient consultation") or the middle level of reported patient involvement (“researcher initiated collaboration”). In fact, the "high involvement" categories that are supposed to be associated with enrollment success are studies that were either fully designed by patients, or were initiated by patients and researchers equally. So, if there is truly a causal relationship at work here, improving enrollment would not be merely a function of adding a patient or two to the conversation.

There are a number of other frustrating aspects of this study as well. It doesn't actually measure patient involvement in any specific research program, but uses just 3 broad categories (that the researchers specified at the beginning of each study). It uses an arbitrary and undocumented 17-point scale to measure "study complexity", which collapses and quite likely underweights many critical factors into a single number. The enrollment analysis excluded 11 studies because they weren't adequate for a factor that was later deemed non-significant. And probably the most frustrating facet of the paper is that the authors share absolutely no descriptive data about the studies involved in the enrollment analysis. It would be completely impossible to attempt to replicate its methods or verify its analysis. Do the authors believe that "Public Involvement" is only good when it’s not focused on their own work?

However, my feelings about the study and paper are an insignificant fraction of the frustration I feel about the public portrayal of the data by people who should clearly know better. After all, limited evidence is still evidence, and every study can add something to our knowledge. But the public misrepresentation of the evidence by leaders in the area can only do us harm: it has the potential to actively distort research priorities and funding.

Why This Matters

We all seem to agree that research is too slow. Low clinical trial enrollment wastes time, money, and the health of patients who need better treatment options.

However, what's also clear is that we lack reliable evidence on what activities enable us to accelerate the pace of enrollment without sacrificing quality. If we are serious about improving clinical trial accrual, we owe it to our patients to demand robust evidence for what works and what doesn’t. Relying on weak evidence that we've already solved the problem ("we've tripled enrollment!") or have a method to magically solve it ("PPI quadrupled enrollment!") will cause us to divert significant time, energy, and human health into areas that are politically favored but less than certain to produce benefit. And the overhyping those results by research leadership compounds that problem substantially. NIHR leadership should reconsider its approach to public discussion of its research, and practice what it preaches: critical assessment of the data.

[Update Sept. 20: The authors of the study have posted a lengthy comment below. My follow-up is here.]
 
[Image via flikr user Elliot Brown.]

Ennis L, & Wykes T (2013). Impact of patient involvement in mental health research: longitudinal study. The British journal of psychiatry : the journal of mental science PMID: 24029538

Every Unhappy PREA Study is Unhappy in its Own Way

“Children are not small adults.” We invoke this saying, in a vague and hand-wavy manner, whenever we talk about the need to study drugs in pediatric populations. It’s an interesting idea, but it really cries out for further elaboration. If they’re not small adults, what are they? Are pediatric efficacy and safety totally uncorrelated with adult efficacy and safety? Or are children actually kind of like small adults in certain important ways?

Pediatric post-marketing studies have been completed for over 200 compounds in the years since BPCA (2002, offering a reward of 6 months extra market exclusivity/patent life to any drug conducting requested pediatric studies) and PREA (2007, giving FDA power to require pediatric studies) were enacted. I think it is fair to say that at this point, it would be nice to have some sort of comprehensive idea of how FDA views the risks associated with treating children with medications tested only on adults. Are they in general less efficacious? More? Is PK in children predictable from adult studies a reasonable percentage of the time, or does it need to be recharacterized with every drug?

Essentially, my point is that BPCA/PREA is a pretty crude tool: it is both too broad in setting what is basically a single standard for all new adult medications, and too vague as to what exactly that standard is.

In fact, a 2008 published review from FDA staffers and a 2012 Institute of Medicine report both show one clear trend: in a significant majority of cases, pediatric studies resulted in validating the adult medication in children, mostly with predictable dose and formulation adjustments (77 of 108 compounds (71%) in the FDA review, and 27 of 45 (60%) in the IOM review, had label changes that simply reflected that use of the drug was acceptable in younger patients).

So, it seems, most of the time, children are in fact not terribly unlike small adults.

But it’s also true that the percentages of studies that show lack of efficacy, or bring to light a new safety issue with the drug’s use in children, is well above zero. There is some extremely important information here.

To paraphrase John Wanamaker: we know that half our PREA studies are a waste of time; we just don’t know which half.

This would seem to me to be the highest regulatory priority – to be able to predict which new drugs will work as expected in children, and which may truly require further study. After a couple hundred compounds have gone through this process, we really ought to be better positioned to understand how certain pharmacological properties might increase or decrease the risks of drugs behaving differently than expected in children. Unfortunately, neither the FDA nor the IOM papers venture any hypotheses about this – both end up providing long lists of examples of certain points, but not providing any explanatory mechanisms that might enable us to engage in some predictive risk assessment.

While FDASIA did not advance PREA in terms of more rigorously defining the scope of pediatric requirements (or, better yet, requiring FDA to do so), it did address one lingering concern by requiring that FDA publish non-compliance letters for sponsors that do not meet their commitments. (PREA, like FDAAA, is a bit plagued by lingering suspicions that it’s widely ignored by industry.)

The first batch of letters and responses has been published, and it offers some early insights into the problems engendered by the nebulous nature of PREA and its implementation.

These examples, unfortunately, are still a bit opaque – we will need to wait on the FDA responses to the sponsors to see if some of the counter-claims are deemed credible. In addition, there are a few references to prior deferral requests, but the details of the request (and rationales for the subsequent FDA denials) do not appear to be publicly available. You can read FDA’s take on the new postings on their blog, or in the predictably excellent coverage from Alec Gaffney at RAPS.

Looking through the first 4 drugs publicly identified for noncompliance, the clear trend is that there is no trend. All these PREA requirements have been missed for dramatically different reasons.

Here’s a quick rundown of the drugs at issue – and, more interestingly, the sponsor responses:

1. Renvela - Genzyme (full response)

Genzyme appears to be laying responsibility for the delay firmly at FDA’s feet here, basically claiming that FDA continued to pile on new requirements over time:

Genzyme’s correspondence with the FDA regarding pediatric plans and design of this study began in 2006 and included a face to face meeting with FDA in May 2009. Genzyme submitted 8 revisions of the pediatric study design based on feedback from FDA including that received in 4 General Advice Letters. The Advice Letter dated February 17, 2011  contained further recommendations on the study design, yet still required the final clinical study report  by December 31, 2011.

This highlights one of PREA’s real problems: the requirements as specified in most drug approval letters are not specific enough to fully dictate the study protocol. Instead, there is a lot of back and forth between the sponsor and FDA, and it seems that FDA does not always fully account for their own contribution to delays in getting studies started.

2. Hectorol - Genzyme (full response)

In this one, Genzyme blames the FDA not for too much feedback, but for none at all:

On December 22, 2010, Genzyme submitted a revised pediatric development plan (Serial No. 212) which was intended to address FDA feedback and concerns that had been received to date. This submission included proposed protocol HECT05310. [...] At this time, Genzyme has not received feedback from the FDA on the protocol included in the December 22, 2010 submission.

If this is true, it appears extremely embarrassing for FDA. Have they really not provided feedback in over 2.5 years, and yet still sending noncompliance letters to the sponsor? It will be very interesting to see an FDA response to this.

3. Cleviprex – The Medicines Company (full response)

This is the only case where the pharma company appears to be clearly trying to game the system a bit. According to their response:

Recognizing that, due to circumstances beyond the company’s control, the pediatric assessment could not be completed by the due date, The Medicines Company notified FDA in September 2010, and sought an extension. At that time, it was FDA’s view that no extensions were available. Following the passage of FDASIA, which specifically authorizes deferral extensions, the company again sought a deferral extension in December 2012. 

So, after hearing that they had to move forward in 2010, the company promptly waited 2 years to ask for another extension. During that time, the letter seems to imply that they did not try to move the study forward at all, preferring to roll the dice and wait for changing laws to help them get out from under the obligation.

4. Twinject/Adrenaclick – Amedra (full response)

The details of this one are heavily redacted, but it may also be a bit of gamesmanship from the sponsor. After purchasing the injectors, Amedra asked for a deferral. When the deferral was denied, they simply asked for the requirements to be waived altogether. That seems backwards, but perhaps there's a good reason for that.

---

Clearly, 4 drugs is not a sufficient sample to say anything definitive, especially when we don't have FDA's take on the sponsor responses. However, it is interesting that these 4 cases seem to reflect an overall pattern with BCPA and PREA - results are scattershot and anecdotal. We could all clearly benefit from a more systematic assessment of why these trials work and why some of them don't, with a goal of someday soon abandoning one-size-fits-all regulation and focusing resources where they will do the most good.

Counterfeit Drugs in Clinical Trials?

This morning I ran across a bit of a coffee-spitter: in the middle of an otherwise opaquely underinformative press release fromTranscelerate Biopharma about the launch of their

Counterfeits flooding
the market? Really?

"Comparator Network" - which will perhaps streamline member companies' ability to obtain drugs from each other for clinical trials using active comparator arms -  the CEO of the consortium, Dalvir Gill, drops a rather remarkable quote:

"Locating and accessing these comparators at the right time, in the right quantities and with the accompanying drug stability and regulatory information we need, doesn't always happen efficiently. This is further complicated by infiltration of the commercial drug supply chain by counterfeit drugs.  With the activation of our Comparator Network the participating TransCelerate companies will be able to source these comparator drugs directly from each other, be able to secure supply when they need it in the quantities they need, have access to drug data and totally mitigate the risk of counterfeit drugs in that clinical trial."

[Emphasis added.]

I have to admit to being a little floored by the idea that there is any sort of risk, in industry-run clinical trials, of counterfeit medication "infiltration".

Does Gill know something that the rest of us don't? Or is this just an awkward slap at perceived competition – innuendo against the companies that currently manage clinical trial comparator drug supply? Or an attempt at depicting the trials of non-Transcelerate members as risky and prone to fraud?

Either way, it could use some explaining. Thinking I might have missed something, I did do a quick literature search to see if I could come across any references to counterfeits in trials. Google Scholar and PubMed produced no useful results, but Wikipedia helpfully noted in its entry on counterfeit medications:

Counterfeit drugs have even been known to have been involved in clinical drug trials.^{[citation needed]}

And on that point, I think we can agree: Citation needed. I hope the folks at Transcelerate will oblige.

Brazen Scofflaws? Are Pharma Companies Really Completely Ignoring FDAAA?

Results reporting requirements are pretty clear. Maybe critics should re-check their methods?

Ben Goldacre has rather famously described the clinical trial reporting requirements in the Food and Drug Administration Amendments Act of 2007 as a “fake fix” that was being thoroughly “ignored” by the pharmaceutical industry.

Pharma: breaking the law in broad daylight?

He makes this sweeping, unconditional proclamation about the industry and its regulators on the basis of  a single study in the BMJ, blithely ignoring the fact that a) the authors of the study admitted that they could not adequately determine the number of studies that were meeting FDAAA requirements and b) a subsequent FDA review that identified only 15 trials potentially out of compliance, out of a pool of thousands.


Despite the fact that the FDA, which has access to more data, says that only a tiny fraction of studies are potentially noncompliant, Goldacre's frequently repeated claims that the law is being ignored seems to have caught on in the general run of journalistic and academic discussions about FDAAA.

And now there appears to be additional support for the idea that a large percentage of studies are noncompliant with FDAAA results reporting requirements, in the form of a new study in the Journal of Clinical Oncology: "Public Availability of Results of Trials Assessing Cancer Drugs in the United States" by Thi-Anh-Hoa Nguyen, et al.. In it, the authors report even lower levels of FDAAA compliance – a mere 20% of randomized clinical trials met requirements of posting results on clinicaltrials.gov within one year.

Unsurprisingly, the JCO results were immediately picked up and circulated uncritically by the usual suspects.

I have to admit not knowing much about pure academic and cooperative group trial operations, but I do know a lot about industry-run trials – simply put, I find the data as presented in the JCO study impossible to believe. Everyone I work with in pharma trials is painfully aware of the regulatory environment they work in. FDAAA compliance is a given, a no-brainer: large internal legal and compliance teams are everywhere, ensuring that the letter of the law is followed in clinical trial conduct. If anything, pharma sponsors are twitchily over-compliant with these kinds of regulations (for example, most still adhere to 100% verification of source documentation – sending monitors to physically examine every single record of every single enrolled patient - even after the FDA explicitly told them they didn't have to).

I realize that’s anecdotal evidence, but when such behavior is so pervasive, it’s difficult to buy into data that says it’s not happening at all. The idea that all pharmaceutical companies are ignoring a highly visible law that’s been on the books for 6 years is extraordinary. Are they really so brazenly breaking the rules? And is FDA abetting them by disseminating incorrect information?

Those are extraordinary claims, and would seem to require extraordinary evidence. The BMJ study had clear limitations that make its implications entirely unclear. Is the JCO article any better?

Some Issues

In fact, there appear to be at least two major issues that may have seriously compromised the JCO findings:

1. Studies that were certified as being eligible for delayed reporting requirements, but do not have their certification date listed.

The study authors make what I believe to be a completely unwarranted assumption:

In trials for approval of new drugs or approval for a new indication, a certification [permitting delayed results reporting] should be posted within 1 year and should be publicly available.

It’s unclear to me why the authors think the certifications “should be” publicly available. In re-reading FDAAA section 801, I don’t see any reference to that being a requirement. I suppose I could have missed it, but the authors provide a citation to a page that clearly does not list any such requirement.

But their methodology assumes that all trials that have a certification will have it posted:

If no results were posted at ClinicalTrials.gov, we determined whether the responsible party submitted a certification. In this case, we recorded the date of submission of the certification to ClinicalTrials.gov.

If a sponsor gets approval from FDA to delay reporting (as is routine for all drugs that are either not approved for any indication, or being studied for a new indication – i.e., the overwhelming majority of pharma drug trials), but doesn't post that approval on the registry, the JCO authors deem that trial “noncompliant”. This is not warranted: the company may have simply chosen not to post the certification despite being entirely FDAAA compliant.

2. Studies that were previously certified for delayed reporting and subsequently reported results

It is hard to tell how the authors treated this rather-substantial category of trials. If a trial was certified for delayed results reporting, but then subsequently published results, the certification date becomes difficult to find. Indeed, it appears in the case where there were results, the authors simply looked at the time from study completion to results posting. In effect, this would re-classify almost every single one of these trials from compliant to non-compliant. Consider this example trial:

• Phase 3 trial completes January 2010
• Certification of delayed results obtained December 2010 (compliant)
• FDA approval June 2013
• Results posted July 2013 (compliant)

In looking at the JCO paper's methods section, it really appears that this trial would be classified as reporting results 3.5 years after completion, and therefore be considered noncompliant with FDAAA. In fact, this trial is entirely kosher, and would be extremely typical for many phase 2 and 3 trials in industry.

Time for Some Data Transparency

The above two concerns may, in fact, be non-issues. They certainly appear to be implied in the JCO paper, but the wording isn't terribly detailed and could easily be giving me the wrong impression.

However, if either or both of these issues are real, they may affect the vast majority of "noncompliant" trials in this study. Given the fact that most clinical trials are either looking at new drugs, or looking at new indications for new drugs, these two issues may entirely explain the gap between the JCO study and the unequivocal FDA statements that contradict it.

I hope that, given the importance of transparency in research, the authors will be willing to post their data set publicly so that others can review their assumptions and independently verify their conclusions. It would be more than a bit ironic otherwise.

[Image credit: Shamless lawlessness via Flikr user willytronics.]

Thi-Anh-Hoa Nguyen, Agnes Dechartres, Soraya Belgherbi, and Philippe Ravaud (2013). Public Availability of Results of Trials Assessing Cancer Drugs in the United States JOURNAL OF CLINICAL ONCOLOGY DOI: 10.1200/JCO.2012.46.9577

Preview of Enrollment Analytics: Moving Beyond the Funnel (Shameless DIA Self-Promotion, Part 2)

Are we looking at our enrollment data in the right way?

I will be chairing a session on Tuesday on this topic, joined by a couple of great presenters (Diana Chung from Gilead and Gretchen Goller from PRA).

Here's a short preview of the session:

Hope to see you there. It should be a great discussion.

Session Details:

June 25, 1:45PM - 3:15PM

• Session Number: 241
• Room Number: 205B

1. Enrollment Analytics: Moving Beyond the Funnel
Paul Ivsin
VP, Consulting Director
CAHG Clinical Trials

2. Use of Analytics for Operational Planning
Diana Chung, MSc
Associate Director, Clinical Operations
Gilead

3. Using Enrollment Data to Communicate Effectively with Sites
Gretchen Goller, MA
Senior Director, Patient Access and Retention Services
PRA

Pediatric Trial Enrollment (Shameless DIA Self-Promotion, Part 1)

[Fair Warning: I have generally tried to keep this blog separate from my corporate existence, but am making an exception for two quick posts about the upcoming DIA 2013 Annual Meeting.]

Improving Enrollment in Pediatric Clinical Trials

Logistically, ethically, and emotionally, involving children in medical research is greatly different from the same research in adults. Some of the toughest clinical trials I've worked on, across a number of therapeutic areas, have been pediatric ones. They challenge you to come up with different approaches to introducing and explaining clinical research – approaches that have to work for doctors, kids, and parents simultaneously.

On Thursday June 27, Don Sickler, one of my team members, will be chairing a session titled “Parents as Partners: Engaging Caregivers for Pediatric Trials”. It should be a good session.

Joining Don are 2 people I've had the pleasure of working with in the past. Both of them combine strong knowledge of clinical research with a massive amount of positive energy and enthusiasm (no doubt a big part of what makes them successful).

However, they also differ in one key aspect: what they work on. One of them – Tristen Moors from Hyperion Therapeutics - works on an ultra-rare condition, Urea Cycle Disorder, a disease affecting only a few hundred children every year. On the other hand, Dr. Ann Edmunds is an ENT working in a thriving private practice. I met her because she was consistently the top enroller in a number of trials relating to tympanostomy tube insertion. Surgery to place “t-tubes” is one of the most common and routine outpatients surgeries there is, with an estimated half million kids getting tubes each year.

Each presents a special challenge: for rare conditions, how do you even find enough patients? For routine procedures, how do you convince parents to complicate their (and their children’s) lives by signing up for a multi-visit, multi-procedure trial?

Ann and Tristen have spent a lot of time tackling these issues, and should have some great advice to give.

For more information on the session, here’s Don’s posting on our news blog.

Can FDA's New Transparency Survive Avandia?

PDUFA V commitments signal a strong commitment to tolerance of open debate in the face of uncertainty.

I can admit to a rather powerful lack of enthusiasm when reading about interpersonal squabbles. It’s even worse in the scientific world: when I read about debates getting mired in personal attacks I tend to simply stop reading and move on to something else.

However, the really interesting part of this week’s meeting of an FDA joint Advisory Committee to discuss the controversial diabetes drug Avandia – at least in the sense of likely long-term impact – is not the scientific question under discussion, but the surfacing and handling of the raging interpersonal battle going on right now inside the Division of Cardiovascular and Renal Products. So I'll have to swallow my distaste and follow along with the drama.

Two words that make us mistrust Duke:
 Anil Potti Christian Laettner

Not that the scientific question at hand – does Avandia pose significant heart risks? – isn't interesting. It is. But if there’s one thing that everyone seems to agree on, it’s that we don’t have good data on the topic. Despite the re-adjudication of RECORD, no one trusts its design (and, ironically, the one trial with a design to rigorously answer the question was halted after intense pressure, despite an AdComm recommendation that it continue).  And no one seems particularly enthused about changing the current status of Avandia: in all likelihood it will continue to be permitted to be marketed under heavy restrictions. Rather than changing the future of diabetes, I suspect the committee will be content to let us slog along the same mucky trail.

The really interesting question, that will potentially impact CDER for years to come, is how it can function with frothing, open dissent among its staffers. As has been widely reported, FDA reviewer Tom Marciniak has written a rather wild and vitriolic assessment of the RECORD trial, excoriating most everyone involved. In a particularly stunning passage, Marciniak appears to claim that the entire output of anyone working at Duke University cannot be trusted because of the fraud committed by Duke cancer researcher Anil Potti:

I would have thought that the two words “Anil Potti” are sufficient for convincing anyone that Duke University is a poor choice for a contractor whose task it is to confirm the integrity of scientific research. 

(One wonders how far Marciniak is willing to take his guilt-by-association theme. Are the words “Cheng Yi Liang” sufficient to convince us that all FDA employees, including Marciniak, are poor choices for deciding matter relating to publicly-traded companies? Should I not comment on government activities because I’m a resident of Illinois (my two words: “Rod Blagojevich”)?)

Rather than censoring or reprimanding Marciniak, his supervisors have taken the extraordinary step of letting him publicly air his criticisms, and then they have in turn publicly criticized his methods and approach.

I have been unable to think of a similar situation at any regulatory agency. The tolerance for dissent being displayed by FDA is, I believe, completely unprecedented.

And that’s the cliffhanger for me: can the FDA’s commitment to transparency extend so far as to accommodate public disagreements about its own approval decisions? Can it do so even when the disagreements take an extremely nasty and inappropriate tone?

• Rather than considering that open debate is a good thing, will journalists jump on the drama and portray agency leadership as weak and indecisive?
• Will the usual suspects in Congress be able to exploit this disagreement for their own political gain? How many House subcommittees will be summoning Janet Woodcock in the coming weeks?

I think what Bob Temple and Norman Stockbridge are doing is a tremendous experiment in open government. If they can pull it off, it could force other agencies to radically rethink how they go about crafting and implementing regulations. However, I also worry that it is politically simply not a viable approach, and that the agency will ultimately be seriously hurt by attacks from the media and legislators.

Where is this coming from?

As part of its recent PDUFA V commitment, the FDA put out a fascinating draft document, Structured Approach to Benefit-Risk Assessment in Drug Regulatory Decision-Making. It didn't get a lot of attention when first published back in February (few FDA documents do). However, it lays out a rather bold vision for how the FDA can acknowledge the existence of uncertainty in its evaluation of new drugs. Its proposed structure even envisions an open and honest accounting of divergent interpretations of data:

When they're frothing at the mouth, even Atticus
doesn't let them publish a review

A framework for benefit-risk decision-making that summarizes the relevant facts, uncertainties, and key areas of judgment, and clearly explains how these factors influence a regulatory decision, can greatly inform and clarify the regulatory discussion. Such a framework can provide transparency regarding the basis of conflicting recommendations made by different parties using the same information.

(Emphasis mine.)

Of course, the structured framework here is designed to reflect rational disagreement. Marciniak’s scattershot insults are in many ways a terrible first case for trying out a new level of transparency.

The draft framework notes that safety issues, like Avandia, are some of the major areas of uncertainty in the regulatory process. Contrast this vision of coolly and systematically addressing uncertainties with the sad reality of Marciniak’s attack:

In contrast to the prospective and highly planned studies of effectiveness, safety findings emerge from a wide range of sources, including spontaneous adverse event reports, epidemiology studies, meta-analyses of controlled trials, or in some cases from randomized, controlled trials. However, even controlled trials, where the evidence of an effect is generally most persuasive, can sometimes provide contradictory and inconsistent findings on safety as the analyses are in many cases not planned and often reflect multiple testing. A systematic approach that specifies the sources of evidence, the strength of each piece of evidence, and draws conclusions that explain how the uncertainty weighed on the decision, can lead to more explicit communication of regulatory decisions. We anticipate that this work will continue beyond FY 2013.

I hope that work will continue beyond 2013. Thoughtful, open discussions of real uncertainties are one of the most worthwhile goals FDA can aspire to, even if it means having to learn how to do so without letting the Marciniaks of the world scuttle the whole endeavor.

[Update June 6: Further bolstering the idea that the AdCom is just as much about FDA's ability to transparently manage differences of expert opinion in the face of uncertain data, CDER Director Janet Woodcock posted this note on the FDA's blog. She's pretty explicit about the bigger picture:

There have been, and continue to be, differences of opinion and scientific disputes, which is not uncommon within the agency, stemming from varied conclusions about the existing data, not only with Avandia, but with other FDA-regulated products. 

At FDA, we actively encourage and welcome robust scientific debate on the complex matters we deal with — as such a transparent approach ensures the scientific input we need, enriches the discussions, and enhances our decision-making.

I agree, and hope she can pull it off.]