Introduction
MIT researchers and collaborators have published a study in Nature Medicine examining how AI explainability tools affect diagnostic accuracy across different types of users. The findings reveal a critical insight: the same AI interface can improve performance for one group while becoming a liability for another, depending on the user’s baseline medical expertise.
The Study at a Glance
The research focused on dermatological diagnosis, a domain where AI tools are already supporting clinicians and increasingly reaching patients through AI-powered search products. Participants were divided into non-experts and primary care providers, and were asked to assess medical images alongside AI predictions.
- Non-experts evaluated whether images of skin moles showed cancer.
- Clinicians performed a broader task: providing a differential diagnosis for dermatological disease.
- Multiple explainability interfaces were tested: prediction with confidence, similar images, heat maps, and LLM-generated explanations.
The Core Finding: Expertise Changes Everything
The study found that every explainability approach improved non-expert accuracy, but the improvement largely came from deferring to the model. Primary care providers performed best when they received an AI prediction without any explanation. This underscores that explainability is not a one-size-fits-all component.
“Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error. We know that both AI and explainability methods can engage automation bias in humans, and this anchoring effect is something that must be accounted for when we design AI systems.” — Marzyeh Ghassemi, MIT
Why Non-Experts Improved — and the Risks
Non-experts benefited from AI assistance, particularly in identifying non-cancerous moles. A fairness-constrained model improved accuracy and reduced diagnostic disparities based on skin tone. However, the performance gain came with a significant risk: non-experts relied heavily on the model’s recommendation. Incorrect model output damaged their performance more than correct output improved it.
“The reason non-expert users are better is because they are more reliant on the models. When the model is wrong, it hurts performance more than it helps performance when the model is right. We were just able to train very good AI models for this setting.” — Marzyeh Ghassemi
The Deference Trap
LLM explanations produced the strongest deference effect. Participants trusted those explanations whether the model output was correct or incorrect. They also found vague or generic explanations more convincing. Users who received LLM assistance reported greater confidence in wrong answers.
The Clinician Advantage
Clinicians did not follow incorrect AI explanations in the same way. They remained resilient even when the system produced an erroneous recommendation. Their strongest performance came from a more limited interface that showed the model’s prediction without an explanation. LLM explanations produced the smallest accuracy improvement for clinicians.
“It really comes down to how each group uses the explanation. A clinician already has a diagnosis in mind and checks the AI against their own training, so a bad explanation gets caught. Meanwhile, a non-expert can use that exact same explanation to form an opinion in the first place, so a plausible, confident-sounding rationale can pull them toward the wrong answer. The same tool ends up being an asset for one user and a liability for another.” — Orson Xu, Columbia University
The Patient Risk
Roxana Daneshjou of Stanford University emphasized the vulnerability of patients with limited medical knowledge.
“These findings are important as patients increasingly turn to AI to help with their health care. Our findings show that those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output.” — Roxana Daneshjou
Timing and Automation Bias
The research also examined when users saw AI assistance. People became more deferential when the system showed an explanation before they had the opportunity to make their own diagnosis. This points to a practical design choice: an interface could ask the user for an initial diagnostic hypothesis, then provide an AI recommendation that surfaces alternative conditions for consideration.
The study found that users who deferred most to AI were also the weakest performers when they completed the task without AI support. These participants may gain the most from model assistance, but they face the greatest risk when the model produces incorrect output.
Implications for Interface Design
The study argues against treating explainability as a standard interface component that works identically for every role. The user’s baseline expertise affects whether an explanation acts as a check on the model or becomes a substitute for independent judgement.
- Clinician tools may need a direct model output that supports review against professional judgement.
- Patient-facing tools require particular care around LLM explanations, especially where the system presents a confident narrative for an incorrect recommendation.
- Designers should consider when to present AI assistance, as pre-hoc explanations increase automation bias.
Conclusion
AI has the potential to improve diagnostic accuracy, but the interface through which it is delivered must be adapted to the user’s expertise. A one-size-fits-all approach to explainability is not only ineffective but can be harmful, particularly for patients who lack the medical training to critically evaluate a confident-sounding AI explanation. The study provides a strong evidence base for moving toward role-aware and context-aware health AI interfaces.