Testing Transparency with the TEST Act

A quick update on my last post regarding the enormously controversial -- but completely unmentioned -- requirement to publicly report all versions of clinical trial protocols on ClinicalTrials.gov: The New England Journal of Medicine has weighed in with an editorial strongly in support of the TEST Act. 

NEJM Editor-in-Chief Jeffrey Drazen at least mentions the supporting documents requirement, but only in part of one sentence, where he confusingly refers to the act "extending results reporting to include the deposition of consent and protocol documents approved by institutional review boards." The word "deposition" does not suggest actual publication, which the act clearly requires. 

I don't think this qualifies as an improvement in transparency about the impact the TEST Act, as written, would have. I'm not surprised when a trade publication like Center Watch recycles a press release into a news item. However, it wouldn't seem like too much to ask that NEJM editorials aspire to a moderately higher standard of critical inquiry.

Public Protocols? Burying the lede on the TEST Act

Not to be confused with the Test Act.
(via Luminarium)

4 Democratic members of Congress recently co-sponsored the TEST (Trial and Experimental Studies Transparency) Act, which is intended to expand the scope of mandatory registration of clinical trials. Coverage so far has been light, and mainly consists of uncritical recycling of the press release put out by congressman Markey’s office.

Which is unfortunate, because nowhere in that release is there a single mention of the bill’s most controversial feature: publication of clinical trial "supporting documents", including the patient’s Informed Consent Form (ICF) and, incredibly, the entire protocol (including any and all subsequent amendments to the protocol).

How Rep. Markey and colleagues managed to put out a 1,000-word press release without mentioning this detail is nothing short of remarkable. Is the intent to try to sneak this through?

Full public posting of every clinical trial protocol would represent an enormous shift in how R&D is conducted in this country (and, therefore, in the entire world). It would radically alter the dynamics of how pharmaceutical companies operate by ripping out a giant chunk of every company’s proprietary investment – essentially, confiscating and nationalizing their intellectual property. 

Maybe, ultimately, that would be a good thing.  But that’s by no means clear ... and quite likely not true. Either way, however, this is not the kind of thing you bury in legislation and hope no one notices.

[Full text of the bill is here (PDF).]

[UPDATE May 17, 2013: Apparently, the irony of not being transparent with the contents of your transparency law was just too delicious to pass up, as Markey and his co-sponsors reintroduced the bill yesterday. Once again, the updated press release makes no mention of the protocol requirement.]

Mind the Gap

Modern clinical trials in the pharmaceutical industry are monuments of rigorous analysis. Trial designs are critically examined and debated extensively during the planning phase – we strain to locate possible sources of bias in advance, and adjust to minimize or compensate for them. We collect enormous quantities of efficacy and safety data using standardized, pre-validated techniques. Finally, a team of statisticians parses the data (adhering, of course, to an already-set-in-stone Statistical Analysis Plan to avoid the perils of post-hoc analysis) … then we turn both the data and the analysis over in their entirety to regulatory authorities, who in turn do all they can to verify that the results are accurate, correctly interpreted, and clinically relevant.

It is ironic, then, that our management of these trials is so casual and driven by personal opinions. We all like to talk a good game about metrics, but after the conversation we lapse back into our old, distinctly un-rigorous, habits. Example of this are everywhere once you start to look for them: one that just caught my eye is from a recent Center Watch article:

Survey: Large sites winning more trials than small

Are large sites—hospitals, academic medical centers—getting all the trials, while smaller sites continue to fight for the work that’s left over?

That’s what results of a recent survey by Clinical Research Site Training (CRST) seem to indicate. The nearly 20-year-old site-training firm surveyed 500 U.S. sites in December 2010, finding that 66% of large sites say they have won more trials in the last three years. Smaller sites weren’t asked specifically, but anecdotally many small and medium-sized sites reported fewer trials in recent years.

Let me repeat that last part again, with emphasis: “Smaller sites weren’t asked specifically, but anecdotally…”

At this point, the conversation should stop. Nothing to see here, folks -- we don’t actually have evidence of anything, only a survey data point juxtaposed with someone’s personal impression -- move along.

So what are we to do then? I think there are two clear areas where we collectively need to improve:

1. Know what we don’t know. The sad and simple fact is that there are a lot of things we just don’t have good data on. We need to resist the urge to grasp at straws to fill those knowledge gaps – it leaves the false impression that we’ve learned something.

2. Learn from our own backyard. As I mentioned earlier, good analytic practices are pervasive on the executional side of trials. We need to think more rigorously about our data needs, earlier in the process.

The good news is that we have everything we need to make this a reality – we just need to have a bit of courage to admit the gap (or, occasionally, chasm) of our ignorance on a number of critical issues and develop a thoughtful plan forward.