Hospitals are putting AI chatbots in doctors’ hands. But does the technology improve diagnostic reasoning? A new randomized trial suggests that access alone does not.

Researchers at Stanford, Beth Israel Deaconess Medical Center, and the University of Virginia tested this question. Fifty doctors took part. About half were attending physicians, and the rest were residents still in training. Most worked in internal medicine.

Next, researchers randomly split the doctors into two groups. One group used ChatGPT Plus along with usual tools like UpToDate and Google. Meanwhile, the other group stuck with the usual tools. Each doctor had 60 minutes to work through up to six real patient cases.

Then experts graded each response. The score looked at diagnostic reasoning, not just the final answer. For example, it also covered each doctor’s list of possible conditions, the clues for and against each one, and the next steps they would take.

The result was close. Doctors with the chatbot had a typical score of 76%. Doctors without it scored 74%. That gap was too small to rule out chance. In addition, both groups took about the same time on each case.

The Chatbot Outscored Doctors on Diagnostic Reasoning

The surprise came next. Researchers also ran the chatbot alone on the same cases. It had a typical score of 92%. In fact, that beat the doctors using the usual tools by 16 percentage points. This pattern is not new, because AI has outperformed ER doctors before.

So the tool performed well, but the doctors who used it did not. Why not?

The authors point to a few possibilities. First, the doctors received no training on how to ask the chatbot good questions. Second, the study did not require them to use it in any set way. Therefore, the researchers suggest that training doctors, or building the right questions into daily routines, may help.

However, the authors warn that these results do not mean AI should diagnose patients without a doctor’s oversight. After all, the cases were hand-picked, and the study did not test real patient interviews.

Diagnostic errors are common, and the authors note they contribute to substantial patient harm. Yet many health systems now offer AI chatbots to doctors, often with little or no training. This trial shows that handing doctors the tool does not automatically improve their diagnostic reasoning.

For patients, the takeaway is clear. A doctor with an AI tool is not automatically a better diagnostician. Still, patients deserve to know how their care team used the tools in front of them, especially when a diagnosis goes wrong.