Study reveals gaps in AI medical device validation

A recent study reveals that nearly half of AI-based medical devices approved by the US Food and Drug Administration (FDA) have not been trained on real patient data. Of 521 devices examined, 43% lacked published clinical validation, raising concerns about their effectiveness in real-world settings.

The study highlights that only 22 of these devices were validated through randomised controlled trials, considered the ‘gold standard’ for clinical testing. Some devices relied on ‘phantom images’ instead of real patient data, while others used retrospective or prospective validation methods. Researchers emphasise the importance of conducting proper clinical validation to ensure these technologies are safe and effective.

Researchers hope their findings will prompt the FDA and the medical industry to improve the credibility of AI devices by conducting and publishing clinical validation studies. They believe enhancing these processes’ transparency and rigour will significantly impact patient care.

In Australia, similar regulations exist, with the Therapeutic Goods Administration (TGA) requiring AI-based software to provide information about its training data and suitability for the Australian population. Medical devices must also meet general clinical evidence guidelines to ensure safety and effectiveness.