Physicians Fail to Detect Flawed AI Treatment Recommendations Even When Evidence Contradicts Them, New Study Finds

A new study has demonstrated that physicians often fail to notice when artificial intelligence systems provide flawed treatment recommendations, even when clear evidence of the error is presented directly to them. The research highlights a persistent problem known as automation bias, in which humans tend to place excessive trust in algorithmic outputs that appear objective and computationally derived.

In the experiment, 223 physicians anonymously participated in a series of online scenarios. They were asked to imagine treating patients with a rare disease and to decide whether to administer an experimental therapy whose effectiveness had not yet been confirmed. Before making their decisions, participants received recommendations from an AI system that divided patients into groups based on predicted likelihood of benefit. After selecting patients for treatment, the doctors reviewed actual treatment outcomes and were asked to evaluate the reliability of the AI suggestions.

Researchers deliberately designed the experiments so that the AI predictions would conflict with real results. In the first series, the therapy produced a uniform moderate effect across all patients, yet the algorithm created the false impression that certain groups benefited more than others. In the second series, the treatment was completely ineffective for everyone, but the majority of participants still did not conclude that the therapy offered no benefit at all.

Even after reviewing the outcome data, most doctors continued to view the AI as a reliable source of guidance. They found it difficult to change their initial opinions when new evidence contradicted the algorithm’s conclusions. This effect persisted regardless of the physicians’ level of medical experience, showing that human oversight alone does not automatically catch AI mistakes.

The study carries important implications for the growing use of AI tools in medicine. Such systems are already being deployed or tested to assess complication risks, select treatment strategies, and identify patients requiring closer monitoring. While these tools are intended only as decision-support aids, the research shows that flawed recommendations can still influence clinical choices, potentially leading doctors to withhold effective treatments or administer ineffective ones.

Because the study was conducted in a simulated environment rather than real clinical settings, researchers could precisely control conditions and know exactly where the algorithm was wrong. The authors conclude that simply warning users about possible errors is insufficient. They recommend implementing structured procedures such as requiring an independent evaluation of each case before the AI suggestion is shown, mandating written explanations for agreement with the algorithm, and conducting regular reviews of cases where AI predictions were not confirmed by actual outcomes.