7 min reading time

Former $MELA CEO Dr. Joseph Gulfo’s book “Innovation Breakdown” is so enlightening (and well-written) that I invited him to join the 10x faculty in May.

Was his experience extreme, biased, or common?

BACKGROUND
• Each year, 150,000+ Americans are diagnosed with melanoma, the most aggressive skin cancer.

• Melanoma is the only cancer you can “see coming” and diagnose in time. But diagnosis is so difficult even dermatologist Liz Tanzi didn’t have her mole biopsied. A year later, she had Stage 1 melanoma.

• Mela Sciences’ MelaFind (now on the market) performs a multi-spectral analysis and gives dermatologists a highly sensitive tool to detect melanoma at a curable stage.

MELA’s EXPERIENCE
• MELA secured a rare binding protocol agreement, which is an institutional commitment from FDA, in 2004, so they expected clearance within 180 days of PMA submission.

• MELA completed the largest prospective clinical trial ever performed in melanoma detection and met every endpoint set with FDA in the binding agreement.

• During the PMA review, FDA indicated favorable opinion and MELA expected an imminent advisory panel.

• Instead, after a new FDA director took over, the very next communication from FDA was a Not Approvable Letter!

• After FDA acquiesced and gave MELA a panel meeting. Three days before the meeting, FDA dumped deceitful and flawed analyses of the MelaFind pivotal trial data.

• In all, it took TWELVE YEARS for MelaFind to be cleared for commercialization.

+++

Dr. Gulfo said, “If something this non-invasive and potentially life-saving can have so much difficulty getting through FDA and small-company public financing, what does that portend for other true scientific breakthroughs?”

+++

Does Dr. Gulfo’s story surprise you? Do you believe his is the outlying case or endemic of FDA? Is it Division dependent?

Joe’s $27 book (only $18 including shipping, code MEDBIZ) at http://medgroup.biz/breakdown

---

Karl Schulmeisters
Principal and Founder at ExStreamVR
I understand it was awful for you. I also read from the record that you did not follow the published guidelines

Robert Christensen
Medical Consultant
Oh yes Karl I was told by Dr. Schultz I could appeal to the Commissioner and when I did I waited for 8 months and then I received a $630,000 CMP. It was criminal and awful.

Karl Schulmeisters
Principal and Founder at ExStreamVR
The FDA should be “too big to challenge”. Not that you should not be able to appeal their decisions via due process, but as with the The Courts, the concept itself of the FDA and its role, should be “too big to challenge”. Otherwise we end up back in snake-oil days

Robert Christensen
Medical Consultant
Oh yes, Joe. You of many, know my story of a corrupted FDA approval process. If others wish to know of the story of my pre-amendment, grandfathered, first of a kind devices to reconstruct the diseased, missing or degenerated TMJ, read my book, FDA You Were WRONG! It tells of a corrupt or likely criminal FDA approval process in which I challenged the FDA and was later even fined $630,000 for possibly 3 questionable MDRs. Oh yes, to challenge, even when you are correct can be disasterous.

Karl Schulmeisters
Principal and Founder at ExStreamVR
The critical two components are the “Training dataset” and the “supervisory control”.

Training dataset is the that set of inputs of both “positive” and “negative” examples that are used to “teach” the system what to diagnose. And particularly for optical based systems, this pretty much means a set of images that have been proven and vetted as such. And because of how Machine Learning works, you ideally want a data set of a couple of hundred thousand examples.

For most medical conditions, such datasets do not exist.

The second aspect is the “supervisory control” – namely how along the iterative “teaching” aspect of the system you identify the “false positives/negatives” and feed that back into the system. Ideally your “test” set is different than your “training set” – so that you can show the actual accuracy of the diagnosis.

so now you are talking about an even larger dataset.

Karl Schulmeisters
Principal and Founder at ExStreamVR
Well I’d challenge the “non-invasive” aspect of the MelaFind app. Because the consequences of the diagnosis are “invasive”: False positives result in unnecessary surgery, false negatives result in the cancer becoming far more advanced before actual detection.

Both are outcomes with potentially fatal or injurious consequences.

OTOH, an intrusive device for a new kind of ortho surgery, or even a new kind of drug eluting stent relies on MD skills and expertise in application.

So I think this speaks less to innovation and a lot more to the fairly high barrier of proof that the FDA is going to require for “big data”/”automated analytics” based diagnostics. And this is not necessarily a bad thing. I know a fair bit about “Machine Learning” (which is what these systems mostly are) and have friends who have done things like built the Child Pornography image detection software for one of the large search engines.

Karl Schulmeisters
Principal and Founder at ExStreamVR
Hence why its so hard to prove to the FDA that a product like Mea actually works.

Finally, the number of folks who understand how this Big Data/Machine Learning stuff works, is a very very limited and very expensive group of folks. So its unlikely the FDA will have a great deal of expertise in this area in the near term.

Joseph Gulfo
Author, Innovation Breakdown: How the FDA and Wall Street Cripple Medical Advances at Post Hill Press
We need a strong FDA to make sure that products are safe and effective. Mission creep and changing the definition of the meaning of safe and effective has resulted in the current system. My goal in writing Innovation Breakdown is to start a campaign and advocate for returning the FDA to its appropriate role as a public health agency and removing it from making private health decisions, for example, clinical utility and benefit risk, which are best left to the medical ecosystem and doctors and patients to make. I hope that you read the book and get others to do so, as well…the future of medical innovation is at stake. Thank you.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Benjamin Ghanoongooi: To answer your question YES. I have exoerienced it first hand. FDA now days is another washinachine for the big Medical Device manufaceurers who can get away by biying their approval process. visit my post Code of Conduct and if you like connect and i will send you hard copy documents to assert my statement .

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Rick Lifsitz: Do we really believe that the FDA misused the data? I was working on imaging technology for diabetic foot ulcers at roughly the same time.

Maybe it was misinformation by the FDA, but my understanding was that the sensitivity and specificity were not great at the time. Medical devices need to aid a clinician in decision making or they might as well roll the dice.

I didn’t read the book, but imaging technology has evolved greatly since then and our ability to get a “strong signal” from the skin has as well.

Let’s be careful not to throw the baby out with the bathwater. It makes great headlines, but I’d really love to know the facts. What was the “real” sensitivity and specificity in 2004?

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Benjamin Ghanoongooi: Steve, can you send me the link or information please thanks.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Steve Zweig: Somewhat similar experience. I wrote a chapter in “Principles and Practice of Point-of-Care Testing” about this.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Benjamin Ghanoongooi:
In my humble opinion FDA is a joke!!! They are as much criminal as the corporation they protect such as Boston Scientific. Visit my post Code of Conduct and see for yourself the truth behind all the criminal activities conducted by this criminal organization called Boston Scientific not only I can prove and assert to all the facts I have audio and documents to substantiate all the information in my post. And unfortunately for those of you who tend to believe that FDA is on your side it is not since I have even documents from FDA local inspectors and TUV which is another criminal organization and anti-Semitic who performed investigation and still for profit put human life at risk. Boston Scientific as you all know it to be is nothing but a racist organization and criminal as it gets. If any one of you care for its profession and have integrity and believe in doing what’s right, will see my post and comment with your professional opinions.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Paul Manberg PhD: In my experience, there are always two sides to every FDA review horror story. It is in FDA’s DNA to be risk averse and suspicious of potential industry data mining for most favorable interpretations of study results, probably for good reason. I believe it has gotten worse in recent years because it only takes one reluctant member of the review team to derail a submission, and inexperienced middle managers are less likely to overrule that one entrenched reviewer as they did prior to the whistleblower episode.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Kathleen Chowder at https://www.linkedin.com/in/katcrowder:
Looking at data in a completely wrong way is unfortunately something the FDA does all too frequently. I know this is addressing only one piece of the trials and tribulations Dr. Gulfo went through, but it’s important to have your own statistical experts to refute the agency’s statisticians when they get an idea in their head.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Bill Peatman: In answer to the question, “Does this surprise you.” Nothing surprises me in this business anymore–the good that gets blocked by the system, or the bad stuff that sneaks through the system.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Alison Bourne: What a nightmare for this inventor, I really feel for him. People need to understand that when you are an inventor, time is money and wasted opportunities to get your innovation being used by those who could benefit from it.

Joe Hage
🔥 Find me at MedicalDevicesGroup.net 🔥
From Howard Merry: I followed Dr. Gulfo all the way through FDA. Since they were publicly traded We used them as a comp. I can validate Dr. Gulfo’s experience. Just yesterday I thought about reaching out to him and Rosemary Crane. MELA seems effectively dead now that they’ve been reversed merged into Xtrac/Vtrac. The new CEO said (IIRC) in the last earnings call that the MELA name will be dropped and the product likely discontinued based on costs.

Gordon McKenzie
CEO at GBF Strategy Ltd
Interesting article, perhaps rather oversimplified.

As I understand it, and I would be happy to be corrected, Mela’s evidence demonstrated that their device worked a little better than a relatively inexperienced dermatologist working with less than ideal data (images on screen). It was an improvement, but not a massive one. MM kills, so any improvement is worthwhile, but cost and biopsy rates (absolute and FP) do matter in the real world.

More problematically, the need to have 100% sensitivity (so that they didn’t miss anything) drove their specificity down lower than it would have had to be if it had genuinely been [used as] a guidance tool to help doctors, rather than a simple yes/no. Mela’s later moves to include images or numeric scales supports this. Doctors and the FDA were understandable wary of a ‘black box’ result.

I do agree that Mela were treated harshly, but the situation was more nuanced than at first glance, with real questions around clinical benefit.

Karl Schulmeisters
Principal and Founder at ExStreamVR
Gordon I think you raise an interesting question –
>>Mela’s later moves to include images or numeric scales supports this. Doctors and the FDA were understandable wary of a ‘black box’ result.<< that in turn opens up another can of worms known as the Automation Bias Cognitive Fallacy https://en.wikipedia.org/wiki/Automation_bias namely that when we have an automated system, we tend to over-rely on its results. Where I “decision god” I would require any automated system to be significantly better than the 2SD distribution of practitioners/specialists can do on their own. Because that’s the group that is going to let their skills and decision making lapse when they start using such a device.
.
And it could well be that the FDA took a similar approach in their “harshness”. Disruptive technology has long struggled not just in the Healthcare field. And I suggest rightly so.