Study questions reliability of AI medical guidance
Research shows AI chatbots struggle with medical advice, matching search engines in accuracy and raising safety concerns.
AI chatbots are not yet capable of providing reliable health advice, according to new research published in the journal Nature Medicine. Findings show users gain no greater diagnostic accuracy from chatbots than from traditional internet searches.
Researchers tested nearly 1,300 UK participants using ten medical scenarios, ranging from minor symptoms to conditions requiring urgent care. Participants were assigned to use either OpenAI’s GPT-4o, Meta’s Llama 3, Command R+, or a standard search engine to assess symptoms and determine next steps.
Chatbot users identified their condition about one-third of the time, with only 45 percent selecting the correct medical response. Performance levels matched those relying solely on search engines, despite AI systems scoring highly on medical licensing benchmarks.
Experts attributed the gap to communication failures. Users often provided incomplete information or misinterpreted chatbot guidance.
Researchers and bioethicists warned that growing reliance on AI for medical queries could pose public health risks without professional oversight.
Would you like to learn more about AI, tech and digital diplomacy? If so, ask our Diplo chatbot!