The FDA, statistics and fundamentalism

Yesterday the FDA panel heard and voted on various matters that may or may not lead to approval of Lemtrada, a Multiple Sclerosis drug. [The press is now reporting in a way that makes you think approval is likely…]

Lemtrada had various features that made it hard – if not impossible – to design a proper double-blind test of its effectiveness. [Both Lemtrada and the counter-factual drug Rebif have substantial and identifying side-effects. The patient would know which drug they were on.]

Instead Sanofi/Genzyme designed a test in which the patients were not blind as to which drug they were taking however the people who rated their MS symptoms were blind. [This is the so-called “rater-blind” test.]

On Friday the FDA staff put out an advisory note on Lemtrada which (a) noted the bizarrely nasty side-effects of the drug, and (b) noted some anomalies in the statistics in which Sanofi/Genzyme presented to support the application for approval. This staff note was widely reported (see Bloomberg for an example) and it was thought that the drug may not be approved.

Safety was no surprise. The statistical anomalies were.

Today the FDA Panel heard the case. The best coverage came from Twitter – and I am grateful to Sarah Karlin for her wonderful Twitter account. The panel voted on many things, however I want to highlight just a couple:

Question 1: are the trials adequate and well-controlled?

Vote: 11 no, 6 yes. One abstained.

Question 2: Has the applicant provided substantial evidence of effectiveness of alemtuzumab [Lemtrada] for the treatment of patients with relapsing forms of MS?

Vote: 12 yes, 6 no.

On question #2, after the panel voted, the FDA staff asserted that if you voted no on question #1, you can't vote yes on #2.

The committee chair said they voted yes based on gut feeling which I am not sure is adequate – but the FDA staff assertion is wrong and I believe incompetent. It is perfectly acceptable to say that the trials are not "well-controlled" but there is substantial evidence of effectiveness. I present to you, yet again, the famous "parachute paper".

In that paper Gordon Smith and Jill Pell observe (with appropriate irony) that there has never been a well controlled double-blind study of parachutes as a device to "prevent death and major trauma related to gravitational challenge". Indeed because an adequate and well-controlled test is a requirement for approval parachutes are not approvable.

This is bunk.

We know why a parachute works.

We understand the physics well.

A single person who jumps out of an airplane and lands unharmed with a parachute is a convincing test that parachutes are effective.

Bluntly, you only need double-blind stats when you don't understand the physics.

The FDA staff have (as parodied in the parachute paper) raised their statistical mystique to a cult – as if it were the only valid form of epistemology. But there are other ways of knowing, they are more valid (as per the parachute example) and there is a deep underlying incompetence in the FDA cult of double-blind statistics and in the assertion highlighted above.

We are living in an age where it is possible to have deep knowledge as to how the body works. We know for instance there is a gene which codes for a protein which is necessary in the metabolism of certain selective serotonin reuptake inhibitor drugs and the presence or absence of that gene has effects. Very small sets of statistics that confirm the effects would be valid epistemology just like a very small set of statistics about people who jump out of planes with parachutes would enable you to conclude that parachutes are a good idea if you want to jump out of planes.

I read the statistics paper that the FDA staff put out on Lemtrada. It was not fun reading. It also convinced me that we should be circumspect about both our knowledge and our methods of determining what we know.

Circumspection about epistemological process however is hard for a fund manager. It is even harder for a government department.

John

PS. I wrote that an interest in epistemology would be used as a criteria in assessing job applicants at Bronte Capital. I was not kidding.

Disclosure: there is a traded contingent value right whose value depends on approval for and sales of Lemtrada [ticker: GCVRZ]. We own a small quantity. We also roughly doubled the position after the FDA staff paper was published.