• Posted August 5, 2025

How Accurate Is ChatGPT When Asked About Your Health?

ChatGPT is likely to be hit-or-miss when it comes to figuring out symptoms for a particular illness, a new study says.

The AI program has 49% to 61% accuracy when it comes to identifying symptoms associated with specific diseases, researchers reported recently in the journal iScience.

This might have to do with how AIs like ChatGPT are trained, said lead researcher Ahmed Abdeen Hamed, a research fellow in systems science and industrial engineering at Binghamton University, State University of New York.

ChatGPT is a large language model, a type of AI that learns by analyzing vast amounts of text. But it might be tripping over the medical jargon used by doctors, Hamed said in a news release.

“ChatGPT uses more of a friendly and social language, because it’s supposed to be communicating with average people,” Hamed said. “In medical literature, people use proper names.”

For example, ChatGPT stumbles when matching “sad mood” to “depression,” “facial redness” to “inflammation,” or “seizures” to “febrile convulsion,” the study says.

“The LLM is apparently trying to simplify the definition of these symptoms, because there is a lot of traffic asking such questions, so it started to minimize the formalities of medical language to appeal to those users,” Hamed said.

On the other hand, ChatGPT was great at identifying things for which the medical jargon matches everyday lingo, such as disease terms, drug names and genetic information, researchers said.

Hamed had thought ChatGPT would have “at most 25% accuracy” when tested on those matters.

But the AI was 88% to 97% accurate at identifying disease terms, 90% to 91% accurate in understanding drug names, and 88% to 98% accurate at assessing genetic information, results show.

“The exciting result was ChatGPT said cancer is a disease, hypertension is a disease, fever is a symptom, Remdesivir is a drug and BRCA is a gene related to breast cancer,” Hamed said. “Incredible, absolutely incredible!”

Hamed said he is investigating AI’s ability to answer medical questions, because many are turning to the programs for quick advice.

“People talk to ChatGPT all the time these days, and they say: ‘I have these symptoms. Do I have cancer? Do I have cardiac arrest? Should I be getting treatment?’” Hamed said.

“It can be a very dangerous business, so we wanted to see what would happen if we asked these questions, what sort of answers we got and how these answers could be verified from the biomedical literature,” Hamed said.

Hamed’s goal is to expose these flaws so the models can be adjusted for better responses.

“If I am analyzing knowledge, I want to make sure that I remove anything that may seem fishy before I build my theories and make something that is not accurate,” he said.

For example, ChatGPT was completely flummoxed when asked questions about data stored in GenBank, a National Institutes of Health database that assigns a number to every identified DNA sequence, such as NM_007294.4 for the breast cancer gene BRCA1.

When asked for these numbers as part of its testing on genetic info, ChatGPT just made them up out of whole cloth, a phenomenon known as “hallucinating,” Hamed said.

“Maybe there is an opportunity here that we can start introducing (medical jargon) to the LLMs to provide much higher accuracy, get rid of all the hallucinations and make these tools into something amazing,” he said.

More information

The Cleveland Clinic has more about AI in health care.

SOURCES: Binghamton University, State University of New York, news release, June 20, 2025; iScience, June 20, 2025