Thoughts on medical device clinical studies, especially for regulatory approval

Active Retired

August 2015

2 min reading time

I was prepping for a guest lecture on “clinical studies for medical devices” for a graduate-level class on medical technology regulatory affairs. My career has spanned more than 30 years on this topic.
I was striving for some learned wisdom beyond just technical and process expertise. What do people here think of these musings?
1. Medical device clinical studies are designed by optimists, and interpreted by pessimists.
2. You always wish you have more than you got: more patients, more follow-up, more or different endpoints, etc.
3. In the same way you cannot inspect quality into a product, you cannot wash a compromised clinical study dataset to make it better. Clean garbage is still garbage.
4. Prospective beats retrospective. Controlled beat uncontrolled, and randomized beats any other kind of control. Superiority designs beat non-inferiority designs. Clinical endpoints beat surrogate endpoints.
5. Death is the best clinical endpoint. It only happens once per patient, so the statistics are easy to understand. It’s also cheap and easy and cheap to measure. And nobody argues with you about whether the patient is actually dead or alive. And the hardest of hard clinical endpoints is in-hospital death: if you die in an hospital, you really have to work at it.
6. Meta-analyses are in, and literature reviews are out. Don’t even. But there is a big difference between a patient-level and publication-level meta-analysis. .
7. A Bayesian analysis can cover a multitude of sins. Or expose them all.
8. The surest way to cure a disease is to open a study of it: suddenly all the patients disappear!
9. Patient enrollment does not eat up most of the calendar time a study needs. Instead it is site identification, site approval (IRB, contracts), patient follow-up, closing the database, and analysis/interpretation.
10. Thirty years ago people thought the only value of regulatory clinical studies for medical devices was to get approval to play in the market. Some people still think that. They are short-sighted fools. Such clinical studies also create KOLs, clarify physician training, seed the market with eager users, provide favorable publications to create interest, help obtain favorable reimbursement, etc. etc. Any good regulatory clinical study is designed and executed with all these other goals in mind and fully supported; not just the narrow goal of getting a regulatory approval.
11. It is foolish to estimate the cost of a device clinical study on a per-patient-enrolled basis. Every patient enrolled costs money, sure, but the majority of the overall costs are based not on the number of patients, but on all the other things needed that cost the same whether you enroll 50 total patients, or 1,000: database design, protocol development, data capture systems, site monitoring, CECs and DSMBs, data processing, etc.
12. I am exasperated when non-clinical science people ask “what do the data say?” Data say nothing. Observations are recorded to make data, data are computerized to make databases, databases are analyzed to make conclusions, conclusions are interpreted to aid in decision-making. Only the last two of these is what the questioner is actually getting at. But they fail to recognize what it takes to get there, and all the pitfalls that line the way.

source: https://www.linkedin.com/groups/78665/78665-6036649858931179523

Marked as spam

Posted by Jim Gustafson

Asked on August 10, 2015 12:00 am

57 views

	Private answer Nice! Marked as spam Report spam
	Private answer Julie Omohundro #2 - Please no more endpoints! Keep it focused on what you really need to know; avoid wouldn't it be nice to know. Above all, make sure the endpoints will support the claims you want to make about the device. #3 - But, but...this is why God invented statistics, no? Or maybe it was that other guy, lol. #4 - I come back to making sure your results will support your claims. I have seen too many low-budget clinical studies that left marketing hamstrung, often being able to say only that "it is better than nothing." When will people learn that a bargain basement study that doesn't support your claims is way too expensive? #5 - I'm afraid what you really have to work it is dying of whatever brought you there. Once you get there, nosocomials and errors make it much easier than it ought to be. :( #6 - Literature reviews are actually pretty popular internationally, I think, although for a Class III, not so much. #9 - But patient enrollment does determine whether you stay on target with your project timelines, I think, because it's the least predictable consumer of time. And if you believe what a site tells you about how many patients they can enroll per month, I would like to talk to you about a sweet little real estate deal I have going in Florida. #10 - Agreed!! Regulatory should be the tail, not the dog. It's weird to be in Regulatory and always the one on the team that thinks Regulatory is the last thing you should be worried about. You do a good study, the Regulatory side will take care of itself. You do a bad one, and nothing can save you. And if you don't know the difference, seek your future elsewhere, please. #12 - Sigh. I guess I'll settle for any sign that data are needed to say anything. I remember attending a meeting where the hot topic was how many months of FU were needed to answer a particular research question. Someone summarized the argument as "some people think you need 12 months, some think you need 6 months, others think you only need 3 months...and then some people seem to know the answer before the study has even started." I get exasperated when people tell me we need to do a study to show that the product is safe and effective. Oh noes! We need to do a study to FIND OUT if it is safe and effective! Marked as spam Report spam
	Private answer Deborah E. Parker Well put. Marked as spam Report spam
	Private answer Concise and easy to read. This will be a great lecture. Marked as spam Report spam
	Private answer Great points and very concise! some people do understand the struggle! Marked as spam* Report spam
	Private answer Larry Stevens, RAC The endpoint must prove not just efficacy, i.e. lower blood pressure, but must show clinical utility, i.e. a longer life. In any case, endpoints must be measurable and agreeable to physician users to be clinically significant, i.e. I would prescribe the product. Marked as spam Report spam
	Private answer The frequency an prevalence of the situation described in point 11 can not be underestimated. In small privately funded startup medical companies the shareholders usually want a fixed cost projected for clinical trials. They look to the development or clinical staff "in house experts " (please notice experts is in quotes) to guarantee successful PMA or 520k clinical at a fixed cost and time. The resulting dilemma and drama can be a source of fustration and humor without thoughtful management and open lines of communication. Getting all the assumptions, risk factors and facts can avoid the sudden onset of Murphy ' s Laws during what should be the home stretch of IVD product development. Marked as spam Report spam
	Private answer Michael Lehmicke Nice list. You might want to add something about your experience with regulatory authorities and the way their interpretation of study results often differs from the sponsors. The FDA will comment on a study design, but will never tell you it is good enough until they see the results. Marked as spam Report spam
	Private answer Love this list. I thought I wanted to attend this lecture... but I really want the panel discussion with you and Julie! Re #12 (Julie's last line), I can't tell you how many studies go to the IRB saying that this study will show that the device is safe and effective. Of course, with that mind set, the consent form also says that the purpose of this study is to show how it will help. You cannot presume your outcome. I would add: "Yes, these rules DO apply to you too! This is on my mind because yesterday a PI told me he had to have an exemption from a rule because his intent is to benefit subjects! Sounded good but I could not find that exception to the rule anywhere. Marked as spam Report spam
	Private answer Rene Berard I would disagree that death is the best clinical endpoint. Death may be feared by many, but when it comes down to it, almost no one would be satisfied with merely being alive. In fact, many would prefer death over painful survival, or life without the ability to enjoy it. And many clinical interventions don't seek to prevent death: they seek to improve quality of life. One thing that you brush on, but don't seem to address directly about the "what does the data say part" is the gigantic elephant that's been sitting in the living room of clinical sciences for decades: the use of frequentist statistical analysis defies sound mathematical reasoning. We never come within light years of satisfying the conditions for using the central limit theorem, yet we make inferences as if it applied. We pretend that p-values are the same as the risk of making a type 1 error when rejecting a null hypothesis (that's like pretending that P(A\|B) = P(B\|A)... which is false). We pretend that statistical significance is clinically significant. We turn a blind eye to the fact that averages over large samples simply hide how poorly we understand the underlying phenomenons: that's why we need to use probabilistic models. Marked as spam Report spam
	Private answer Julie Omohundro Erica, thanks for the laugh! Marked as spam Report spam
	Private answer Leslie Wise In all seriousness the most important thing missing is that almost all clinical trials need to be designed for both regulatory bodies: FDA and CMS. No longer is FDA the only Regulatory body that matters. FDA cleared or approved devices that fail to garner reimbursement are bad investments. Marked as spam Report spam
	Private answer Julie Omohundro Rene, I'm very rusty on statistics at this point, but I'm pleased to see someone bring statistical issues to the table. The concept of "clinical significance" is my favorite thing about clinical research. While some may pretend (I would guess far more simply don't know any better) that statistical significance is clinical significance, at least in clinical research the concept is recognized. There is no equivalent concept in the rest of the sciences, i.e, no one speaks of "scientifically significant." And the rest of the sciences are overrun with people treating statistical significance as if it means "important." I don't know enough to agree or disagree with you on whether averages over large samples simply hide how poorly we understand the underlying phenomenon. However, you seem to be coming from the perspective of medical research, where underlying phenomenon are of interest. When industry conducts a pivotal trial of a new medical device, they aren't trying to understand underlying phenomenon; they simply want to show that (sigh) their new device is effective. (And safe, but typically this latter question isn't addressed statistically.) Marked as spam Report spam
	Private answer Beth Ann Fiedler, PhD I would disagree with #6 re: lit reviews. They are still 'in' in terms of foundational/background/historical information and required for clinical support in some meddev classifications. I would say WHO should be doing the lit review, WHAT should be reviewed and WHEN is changing, not IF there should be a lit review. But I would proceed with caution. I am currently doing a book on meddev regulatory requirements. Certain product information is only relevant from about 2011 forward and discrepancies in approval can be based on local lander locations (state) in Germany to how a nation determines if a drug eluting device will undergo scrutiny as a pharmaceutical or a meddev in the EU. Yet once a product undergoes CE Marking in the EU, they can be distributed anywhere in the EU, even if they have not met the clinical parameters or lit review logic in an EU nation that would require different clinical support due to different classification. Better to compensate with something as product moves into a nation where it was not developed, even if it is just a 'lit review'. Marked as spam Report spam
	Private answer Beth Ann Fiedler, PhD TO Michael Lehmicke's point: in addition to sponsor interpretation, you must also consider the experience of Notified Bodies in the EU. Marked as spam Report spam
	Private answer Mahmood Hassan Hi, its really important topic, and very informative as well. Marked as spam Report spam
	Private answer Like your analysis. #1 is funny but unfortunately true... It's an utopia to imagine a proper scientific objective analysis for such topics. There are so much money, personal interest and passion involved in these study. That leads directly to the #12, conclusions are directly related to the study design... With a big unpredictable maze in between. Part about death is funny and true too. This is a great binary measurement point. But when you argue on quality of life improvement, you cannot wait for the patient to die, especially if your product improve significantly people life's :) Thanks for the lecture Have a nice day Marked as spam Report spam
	Private answer Great list! I especially like #8, #9, #10, and #11. Many people don't realize the time or the cost of those activities that create, support and analyze the trial such as protocol development, trial set-up, database creation and management, data integrity services such as monitoring, etc. These costs are necessary regardless of the number of subjects. You may want to add a point about calculation of the number of subjects needed for a trial. I have spoken to many people who want to calculate the number of subjects needed based on their budget instead of the probability of getting useful results. Marked as spam Report spam
	Private answer Dorothy Erlanger Excellent! Point 12 reminded me of a great Theodore Levitt quote: "Data are not information. Information is not meaning. Just as data require processing to become informational, so information requires processing to become meaningful." Marked as spam Report spam
	Private answer Dorothy Erlanger Your item 10 is critically important to product success. Because of all of the market implications, it is imperative to involve key marketing and strategic decision-makers at this stage. If not, they could end up with the wrong centers, wrong KOLs, weak positioning. Yet few medical device companies do this effectively, if at all. Marked as spam Report spam
	Private answer Don't forget that success does not mean insurers will pay Marked as spam Report spam
	Private answer Has been 5 months when this was discussed, what I can tell us nurse, patients and worked on medical device industry: # most important target should be patients security #my personal opinion clinical trials don't add so many value before launch, to do a real one as they way that they should be done, will raise cost of product. #More regulation with will delay access to patients that can be benefit # I will invest on more robust technoloviligance programs and proactive technovigilance programs # this taking in consideration GMP, Quality Programs and all basis that already is in place to register a medical devices . # we should have more investment on analyzing data, not just capturing or generating it. This probably will add more value than clinical trials to register . Marked as spam Report spam
	Private answer Rene Berard Gabriela Garnham you raise some very good points. The last part, in particular: the industry as a whole focuses too much on generating huge amounts of data, and relying on statistical tests to determine whether something is safe and/or effective, and not enough on actually understanding the underlying physiological (or psychological) processes. The result is that we know a lot about how sensitive and specific a diagnostic procedure is or how safe and effective a treatment is "on average" for a given population, but we know very little about how it affects any given patient. We need to pay more attention to individual exceptions. That is the only way to gain a true understanding of any phenomenon. Relativity and quantum mechanics would never have been developed by analyzing large samples of the most common physical phenomena. Only by studying extreme cases did scientists realize that Newtonian mechanics were flawed, and gain improved understanding of underlying physics. Marked as spam Report spam

« Back to Previous Page

Meet your next client here. Join our medical devices group community.