*This post was originally posted on the Context Matters blog*
Context Matters will be exhibiting at Evidence USA this October in Boston. Join us at America's largest and most holistic event on the use of real world outcomes for pharmaceutical products.
Last week we wrote a post with the hopes of opening discussion about the FDA's new "Breakthrough Therapy" designation, and started by looking at how one of the previous FDA fast lanes "Priority" compared to their "Standard" lanes and also to the European Medicines Agency (EMA).
In that analysis, we compared FDA data from 2008 and 2011 on "Priority" and "Standard" lanes and also on EMA data from 2012. The EMA has a different system that emphasizes reviewing all compounds expediently (it's working – EMA's cycle time is less than what it is in the U.S.). Our analysis revealed that over time the FDA's "Priority" lane slowed significantly.
These findings prompted some excellent dialog from those in the industry around our methodology and how we came to these numbers. So, we thought it would be worth going a little deeper into the subject. In this posting, we will start by concentrating exclusively on Priority reviews. In a future post, we will look at Standard reviews.
Before we get started, we should note that the numbers in our previous post "Life in the Fast Track" were initially reported based on FDA Fiscal Year format and should have been converted to Calendar Year. We apologize for this oversight and the numbers in that post have now been updated. The underlying trend of "Priority" slowing down still remains.
What is in a "cycle time"?
One important bit of context is that we focus on a different measure of time than the FDA does. The FDA's "cycle time" is the time it takes to respond to the applicant after any individual submission. While this is a good measure of one aspect of the FDA's performance, we are actually more interested in the length of the whole approval process as this is the time span that is most pertinent to anyone developing drugs. For instance, if a drug goes through multiple cycles (which sometimes happens) the whole process will take the multiple of any single cycle time. For that reason, in our last blog on this topic we reported figures for the total time from initial submission to approval. We will continue to focus on this more comprehensive measure of "cycle time" in this blog.
Are outliers skewing the numbers?
Much of the discussion of our last blog involved "outliers," specifically whether a few unrepresentative extreme values might have thrown off the analysis. Good question. Let's look at the data.
The two charts below show how long every approval took in 2008 and 2011. The first chart shows the total times for 2008 approvals. The second shows the total times for 2011 approvals. Both charts order drugs by total time to approval. Those with the shortest times are on the left; those with the longest times are on the right.
Notice that the last columnâthe longest time in 2011 – is literally off the charts. At 120 months, it won't fit on the scale we're using (if we changed the scale to accommodate this one point, it would be hard to see differences in the other values). This one column is more than twice the next longest time in either year, which fits any reasonable definition of "outlier."
To see whether outliers are unduly influencing our results, we'll remove them and recalculate the average total times in each year. We've identified one outlier, but are there others? There are no definitive criteria to apply, but it does seem that in 2011 the three longest times are unusual – they are all about 4 years or longer while the rest of the times are under 3 years – so we'll eliminate them. In 2008, the longest two times are about 3 years or longer, while all of the remaining times are less than 18 months. We will eliminate the longest two times.
Without these outliers in the mix, average total time to approval rises from 8 months in 2008 to 10 months in 2011. That's a 25% increase in average approval time over just 3 years.
Of course, this result depends on which approval times we define as outliers. We made what seemed like the most reasonable choices, but there is really no good hard and fast rule for defining outliers. Luckily, there is a better way to account for outliers.
Mean or Medianâwhich do you prefer?
To date, we have reported mean times (what people usually refer to when they talk about the "average"). The mean has some nice properties, but it has one drawback in this context. It is sometimes sensitive to extreme values, or outliers.
The median – the middle number in a set of numbers – is less sensitive to outliers than is the mean. All that matters about a number for calculating the median is the number's rank when they are ordered from small to large. In this analysis, even the median total approval time increased from 6 months in 2008 to 8 months in 2011, a 30% increase in approval time.
So while there were some unusually long approval times in both years, and even accounting for this – in two different ways – we see that total approval times increased between the two years.
But that isn't the whole story. In 2008 the FDA approved 17 Priority drugs. Three years later in 2011 it approved 27, an almost 60% increase in the number of drugs approved in the Priority process. If the total number of priority drugs which went through the system increased so much, maybe we shouldn't be surprised that the fast lane got a little backed up.
Watch for a future blog when we delve further into the performance differences of standard review.