Researchers from Stanford, Binghamton, Texas A&M and Indiana argue in Nature Health that harm from chatbot health advice goes untracked.
In one real case, a 60-year-old man was hospitalized with bromide poisoning after following a chatbot's diet suggestion.
The authors want users to see their own health chats and harmful exchanges reported like bad drug reactions.
They also call for malpractice-style liability for chatbots giving medical guidance and for changes to Section 230.
Quick read · 1 min
A man aged 60 wanted less chloride in his meals, asked ChatGPT for help, swapped his table salt for sodium bromide and was hospitalized with paranoia and hallucinations. His doctors only learned days later that a chatbot was involved, and they never saw the conversation.
Researchers from Stanford, Binghamton, Texas A&M and Indiana use that case in a Nature Health paper to argue that harm from AI health advice goes untracked. About one in three US adults had used AI for health information by March 2026, yet chats stay locked with the AI company.
They want people to be able to see their own health chats, with harms reported like bad drug reactions.
They call for independent research access and malpractice-style liability for chatbots.
For now, save any health chat that guides a decision and show it to your doctor.
A 60-year-old man wanted to cut chloride out of his diet, so he asked ChatGPT how. He ended up replacing his table salt with sodium bromide and landed in the hospital with paranoia and hallucinations. It took days of treatment before his doctors even learned a chatbot was involved, and they never saw what it had told him.
That case sits at the center of a new paper in the journal Nature Health about AI health advice. Its warning matters to anyone who has typed a symptom into a chatbot: when the advice goes wrong, there’s usually no record that a doctor, regulator or researcher can check. Mistakes can repeat quietly because nobody is looking.
01
Why harm from AI health advice leaves no trace
Most medical harm leaves paperwork behind. A doctor’s advice goes in your chart. A drug that hurts people triggers adverse-event reports, and as those reports pile up, regulators can spot a pattern and act.
A chatbot conversation works differently, the authors argue, in two ways. First, the exchange is visible only to you and the company running the model, and AI companies don’t give outside researchers a way to study those chats. Second, a fluent, confident answer gives you no hint that something is off. If a bot wrongly tells you a symptom is nothing to worry about, you might never link a diagnosis that came too late back to that chat.
The bromide case shows the problem in practice. Without the original conversation, the man’s doctors couldn’t work out exactly what advice he had followed.
The paper comes from Matthew DeVerna of Stanford University, Kai-Cheng Yang of Binghamton University, Harry Yaojun Yan of Texas A&M University and Filippo Menczer of Indiana University. It is a perspective piece, meaning it builds an argument from published evidence rather than reporting a new experiment. Yang put it bluntly, saying that right now nobody is looking at these errors.
02
How many people now ask chatbots about their health
Plenty. A KFF poll found that by March 2026, about one in three US adults had used AI for health information or advice. Among those users, 65% said a major reason was wanting an answer right away, and 36% said they felt more comfortable asking health questions in private.
The researchers don’t dismiss those reasons. A chatbot can turn a page written for doctors into plain language and answers in the middle of the night If you have no doctor close by, or English is not your first language, it may be the only medical advice you can afford.
The worry is oversight. AI tools used inside hospitals can fall under medical rules and hospital safeguards. General-purpose consumer chatbots often don’t. In the authors’ view, these apps currently treat a question about chest pain no differently from one about when a restaurant opens.
03
The ways chatbot health answers go wrong
The paper lists several failure points:
Hallucination: the model writes smooth, confident statements that are simply false.
Data poisoning: people plant false claims on websites that a model later soaks up in training.
Inconsistency: quality varies between chatbots and even with how you phrase the same question.
Language gaps: answers in languages other than English are much less accurate and consistent, and the people who depend on them most have the fewest alternatives.
You don’t even have to ask a question to be exposed. Grok, the AI built into X, answers users’ questions about posts in public feeds, so its replies on topics like vaccine safety are seen by everyone scrolling past. Some accounts now post mostly AI-made text, images or video, and their confident health claims reach millions. Coordinated groups of AI agents can also make one claim look like a crowd of independent voices agreeing.
Measuring any of this is hard, the authors note, because platforms have pulled back the data access researchers once used to study how content spreads.
04
What the researchers want AI companies and lawmakers to do
Their first ask is simple: you should have a right to see your own health conversations with an AI system. That would let a harmful exchange be rebuilt and shown to a doctor. Those cases should then be reported and investigated the way bad drug reactions are.
Every major AI company already has the technical ability to build such a system, the paper says. But a company checking itself gets to decide what counts as harm, so the authors also want independent researchers to have ongoing access to deployed chatbots, in every language they serve.
Other proposals include:
Labeling AI-generated health content on platforms and holding it back until experts have checked it.
Handling medical questions as riskier than everyday ones, with answers drawn only from material approved by medical societies.
Applying malpractice-style liability to chatbots that give medical guidance.
Changing Section 230, the US law that generally shields online platforms from liability for content posted by others.
Yang argued that companies can’t be relied on to police this themselves, because building such checks cuts against their incentive to make money. He wants a third-party monitoring system instead.
What this means if you use a chatbot for health questions
None of these changes exist yet, so the protection is up to you for now. A few practical habits help:
Save the conversation. If a chatbot’s advice shapes a decision about medicine, diet or symptoms, keep a copy or screenshot so you can show your doctor.
Tell your doctor about it. The bromide case dragged on partly because nobody knew a chatbot was involved.
Don’t swap substances on a bot’s say-so. Replacing a food, supplement or medicine is exactly the kind of change to run past a pharmacist or doctor first.
Be extra careful in other languages. Answers outside English are less reliable.
Treat health posts in your feed with suspicion. A confident AI reply under a viral post isn’t medical advice.
Used this way, a chatbot can still help you understand a diagnosis or prepare questions for an appointment, without becoming the last word on your health.
06
Quick answers
Why can’t doctors see harmful AI health advice?
Chatbot conversations are stored only on the AI company’s platform and your device. Unlike charts or drug reaction reports, there’s no system that shares them with doctors or regulators.
What happened in the sodium bromide case?
After asking ChatGPT for a way to remove chloride from what he ate, a man aged 60 replaced table salt with sodium bromide and was hospitalized with bromide toxicity, including hallucinations and paranoia.
Should I stop using AI for health questions?
Not necessarily. Use it for background and plain-language explanations, keep a record of the chat and check any decision about treatment or diet with a professional.