AI helps in some medical settings and fails in others. A large Swedish trial found AI-supported breast screening detected more cancers, while other studies found a hospital sepsis alert missed about two thirds of cases and chatbots did not help people decide what to do about symptoms. The WHO and NIST warn about false outputs, bias and over-reliance. This page is general information, not medical advice.
01Where is AI helping in medicine?
This page is general information, not medical advice. For your own health, ask a doctor, nurse or pharmacist.
The strongest evidence we found is a randomised trial in Swedish breast cancer screening. About 106,000 women were split between standard screening and AI-supported screening, and AI support found 29% more cancers (338 against 262), cut radiologists' workload by 44% and raised false alarms by about 1%. It was published in The Lancet Digital Health on 4 February 2025 (Lund University summary).
In the United States, the FDA keeps a public list of AI-enabled medical devices it has authorised. The FDA says the list is not comprehensive, and it does not identify any device as built on a large language model. It says it plans to explore ways to tag them (FDA list).
The Swedish trial involved one country and radiologists who reviewed the AI's output. Results may not carry over to other settings or to AI used without a human check.
02What happens when AI gives wrong or invented answers?
NIST gives a healthcare example of the hallucination problem: a false summary of a patient's records could lead a doctor to a wrong diagnosis or treatment (NIST AI 600-1).
A research team tested OpenAI's Whisper speech-to-text system as of 2023. About 1% of transcriptions contained phrases nobody spoke, and roughly 38% of those invented phrases caused concrete harm. Invented content was more frequent for people who pause for long stretches, as some people with aphasia do (Koenecke and others). The authors also warned about uses such as patient notes.
In a randomised study of nearly 1,300 people, those who used chatbots to judge symptoms and choose a next step, such as seeing a GP or going to A&E, did no better than people who used web searches or their own judgement. Users often did not know what details to give, and the answers mixed good and poor advice (NIHR Oxford BRC, February 2026).
03What about prediction tools used inside hospitals?
University of Michigan researchers tested the Epic Sepsis Model on 38,455 hospitalisations. The model flagged only about a third of sepsis cases (33%), missing about two thirds, and it raised alerts for roughly 18% of all patients. Its discrimination score of 0.63 was below the 0.76 to 0.83 reported earlier. The authors concluded that independent testing is needed before such tools are relied on (JAMA Internal Medicine, 2021).
This was one model at one health system in 2018 and 2019. It shows why an outside check on real patients matters, not that every prediction tool performs the same way.
04Can medical AI be biased?
Researchers asked four chatbots nine questions based on race-based formulas and medical misconceptions, five times each. Every chatbot gave at least some answers that promoted race-based medicine or repeated unsupported racial claims, and answers to the same question varied between runs. The authors concluded that such tools could spread debunked, harmful ideas if used in healthcare and called for further evaluation before clinical use (Omiye and others, 2023).
The WHO's January 2024 guidance on large multi-modal models lists risks including false or incomplete outputs, skew in training data by race, ethnicity, sex, gender identity or age, over-reliance by clinicians and patients, and security weaknesses that could expose patient information (WHO).
05What do regulators say, and what can a patient do?
The WHO guidance contains more than 40 recommendations for governments, technology companies and health providers. In the US, the FDA published a draft guidance on 7 January 2025 on lifecycle management and marketing submissions for AI-enabled device software. It says the draft is not final and not for implementation yet (Federal Register notice). NIST adds that false generative AI content is especially important to monitor when it feeds consequential decisions (NIST AI 600-1).
Practical habits that follow from the sources above:
- Use a chatbot to prepare questions for your clinician, not to decide whether you need care.
- For urgent or worrying symptoms, contact a clinician or emergency services.
- Ask your clinic whether AI is used in your care and who checks its output.
- Treat AI-written notes or summaries as drafts, and ask for errors in your record to be corrected.
?What is not known
- We could not confirm whether the FDA's January 2025 draft guidance on AI-enabled device software has since become final.
- The Whisper and sepsis results describe specific versions tested in 2019 to 2023. Newer versions may behave differently.
- The Swedish trial's source is a university news article. Later results on cancers found between screenings were not checked against a primary paper.
QQuestions people ask
Is it safe to ask a chatbot about my symptoms?
The Oxford study found chatbot users decided no better than people who searched the web. Treat a chatbot as a way to prepare questions, and ask a clinician before acting on health advice.
Who checks medical AI tools?
In the US, the FDA authorises some AI-enabled devices and keeps a public list, though it calls the list incomplete. Its guidance on AI device software is still a draft. The WHO has issued recommendations for governments, companies and providers.
Do doctors use AI tools to read scans?
In some settings, yes. The Swedish screening trial used AI support alongside radiologists, and the FDA list names devices it has authorised for use in the United States.
Last checked October 24, 2026. Rules and settings change, so open the source to see the current version.
- AI-supported breast cancer screening: new results suggest even higher accuracy Lund University A university summary of the MASAI randomised trial; we used its trial size, detection, workload and false-alarm figures.
- List of Artificial Intelligence-Enabled Medical Devices US Food and Drug Administration The FDA's public device list; we used its statements about coverage and about large language models.
- NIST AI 600-1, Generative AI Profile National Institute of Standards and Technology NIST's July 2024 profile; we used its healthcare example of a false patient-record summary.
- Careless Whisper: Speech-to-Text Hallucination Harms arXiv (Koenecke and others) A 2024 paper on invented phrases in transcription; we used the rate, the harm share and the finding on speech pauses.
- Study warns of risks in AI chatbots giving medical advice NIHR Oxford Biomedical Research Centre A February 2026 summary of a randomised study in Nature Medicine; we used its design and findings.
- External validation of a widely implemented proprietary sepsis prediction model JAMA Internal Medicine The 2021 University of Michigan study of the Epic Sepsis Model; we used its sensitivity, alert rate and conclusion.
- Large language models propagate race-based medicine npj Digital Medicine A 2023 study of four chatbots; we used its design and main finding.
- WHO releases AI ethics and governance guidance for large multi-modal models World Health Organization The WHO's 18 January 2024 announcement; we used its list of risks and the number of recommendations.
- Artificial Intelligence Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations US Food and Drug Administration (Federal Register) The 7 January 2025 notice of the FDA draft guidance; we used its status and date.
General information, not legal, medical, financial or investment advice. We are not affiliated with any company, agency or standards body named here. See our editorial policy and corrections, or tell us about a mistake at [email protected].