MIT develops safer way to detect harmful AI models

MIT researchers have developed a new auditing method to detect whether generative AI models have been adapted to produce child sexual abuse material without generating illegal content during testing.

The technique was developed with Thorn, a child safety nonprofit focused on protecting children from sexual abuse and exploitation online.

Traditional AI safety testing often involves prompting a model and checking its outputs, but that approach cannot be used for child sexual abuse material, which is illegal to generate in the US and many other jurisdictions.

MIT said the problem has become more urgent as open-source generative AI models become easier to download, adapt and redistribute.

The researchers’ method examines internal changes during fine-tuning, rather than testing the model by generating images.

In tests, the auditing procedure identified model variants adapted to generate child sexual abuse material with 100% accuracy.

MIT said hosting platforms could use the method to flag unsafe models, block uploads or remove harmful adaptations before they spread more widely online.

The researchers also plan to test whether the approach can detect harmful capabilities in a larger set of model variants and in base models before adaptation.

Why does it matter?

The research addresses a serious AI safety blind spot: some harmful model capabilities cannot be tested safely or legally by generating outputs. A non-generative auditing method could give hosting platforms, auditors and law enforcement a safer way to detect models adapted for child sexual abuse material before they are distributed. It also points to a broader governance challenge around open-source generative AI: platforms may need scalable tools to assess harmful adaptations without exposing reviewers to illegal or traumatic content.

Would you like to learn more about AI, tech, and digital diplomacy? If so, ask our Diplo chatbot!