One of the most common criticisms of clinical AI is that it behaves like a black box. The intuitive solution is explainable AI (XAI): show users why a model made its prediction, and they should be able to use it more safely. Divergent impacts of explainable AI for dermatological diagnosis on clinicians versus lay people by Xu et al. (2026) complicates that assumption ¹. An explanation does not merely reveal information about a model. It also changes the person receiving it.

The paper studies how different explanations affect dermatological diagnosis among the general public and primary care physicians (PCPs). Its most important result is not simply that AI improved accuracy. It is that the same natural-language explanation had opposite risks depending on who read it: it increased lay users’ tendency to follow both correct and incorrect AI predictions, while experienced PCPs were more resistant to incorrect advice.

This makes the paper especially relevant to my interest in interpretable and trustworthy medical AI. A model can be explainable in a technical sense while its explanation still makes the combined human-AI system less reliable.

What counts as an explanation?

The authors compared four forms of AI assistance:

• Basic: the predicted diagnosis and model confidence.
• GradCAM: a heatmap showing which image region most influenced the prediction.
• Content-based image retrieval (CBIR): three visually similar reference cases.
• Large language model (LLM): a natural-language rationale describing visual features that supposedly support the prediction.

There is an important distinction in the LLM condition. GPT-4V did not independently diagnose the image. A separately trained image classifier first produced a label. The image and that label were then given to GPT-4V with instructions to explain why the label was correct. If the classifier was wrong, GPT-4V was still asked to justify the wrong answer.

Therefore, the generated text should not be interpreted as a faithful explanation of the classifier’s internal reasoning. It is a post-hoc rationale conditioned on the classifier’s output. This design is useful because it resembles a real failure mode: once an AI system commits to a diagnosis, a fluent language model can construct a plausible story around it whether or not the diagnosis is correct.

GradCAM, CBIR, and LLM explanations also require different kinds of cognitive work. A heatmap must be interpreted, retrieved images must be compared, but an LLM directly connects observations to a conclusion in authoritative prose. That convenience can be helpful, yet it can also make the explanation unusually persuasive.

Two studies, three levels of expertise

The researchers conducted two randomized online experiments using a 4-by-2 between-subjects design. Each participant was assigned one explanation type and one decision order:

• Human-First: make an independent diagnosis, then see the AI and reconsider.
• AI-First: see the AI alongside the image, then reconsider after the AI output is hidden.

Every participant reviewed 12 images and reported a diagnosis and confidence twice.

Study 1 recruited 623 members of the general public without medical backgrounds. They performed a binary melanoma-versus-nevus task. The images were balanced across diagnoses and between lighter skin tones (Fitzpatrick I-IV) and darker skin tones (V-VI). Each participant saw ten correct and two incorrect AI predictions, fixing AI accuracy at 83.3%.

Study 2 recruited 153 PCPs with an average of six years in practice. They entered an open-ended top-three differential diagnosis for images emphasizing atopic dermatitis, pityriasis rosea, Lyme disease, and cutaneous T-cell lymphoma. The AI was correct on an expected 9.5 of 12 cases, or 79.2%. An additional cohort of 320 medical students completed the same task, allowing the authors to examine expertise without directly comparing the easier public task with the harder PCP task.

The paper appropriately warns that Studies 1 and 2 are not directly comparable. The participants, diagnoses, and response formats all differ. The medical-student cohort is therefore important: it provides a comparison on the same task rather than assuming that every difference between the public and PCPs is caused by expertise.

Fairness came from the predictive model, not the explanation

Before studying explanations, the authors trained image classifiers with conditional domain-adversarial neural networks (CDANNs) to reduce performance disparities across skin tones. For the melanoma-versus-nevus model, fairness-constrained training reduced the balanced-accuracy gap from 9.1 to 2.1 percentage points while maintaining similar overall performance. For the main PCP model, the gap decreased from 14.4 to 5.6 points.

Once humans received AI assistance, their own skin-tone-related disparities also narrowed. Among lay participants in the Human-First group, the accuracy gap decreased from approximately 3.3 to 1.7 percentage points. Among PCPs, it decreased from approximately 4.6 to 2.9 points.

However, the authors found no significant interaction between explanation type, decision round, and skin tone. The improvement in equity was mainly attributed to the underlying fairness-constrained prediction model rather than GradCAM, CBIR, or LLM explanations.

This distinction matters. An explanation cannot repair a model whose errors are systematically concentrated in an underrepresented group. Human-AI fairness depends first on the quality and distribution of the model’s predictions. Explanation design determines how users react to those predictions afterward.

LLM explanations amplified lay users’ deference

Among the 335 lay participants assigned to the Human-First workflow, average accuracy increased from 69.7% without AI to 75.8% after AI assistance.

All four assistance types improved overall performance. The numerical improvement was largest for LLM explanations at 7.7 percentage points, followed by CBIR at 6.3, GradCAM at 5.5, and the basic prediction at 4.8. Looking only at the average, natural-language explanations appear to be the best interface.

The average hides the central result. When the AI was correct, the LLM group improved by 13.4 points, more than the basic, GradCAM, or CBIR groups. When the AI was wrong, the LLM group declined by 21.1 points, also the largest deterioration. Compared with the basic condition, the additional harm from an incorrect LLM explanation was statistically significant in the authors’ difference-in-differences analysis.

The LLM therefore did not simply add diagnostic information. It increased AI deference: users were more likely to move toward the AI prediction regardless of whether that prediction was correct. When the model was accurate most of the time, this behavior raised average performance. On the deliberately included error cases, the same behavior became automation bias.

This is why evaluating XAI only with mean accuracy can be misleading. A persuasive explanation can look beneficial when model accuracy is high while making rare failures more difficult for users to catch. In medicine, those failures may be the cases where human oversight matters most.

Expertise changed how explanations were used

The PCP task was substantially more difficult. In the Human-First group, unassisted top-one accuracy was 11.5%, and top-three accuracy was 16.1%. AI assistance increased these measures by 21.5 and 43.5 percentage points, respectively.

Unlike the general public, PCPs did not show a significant decrease in aggregate accuracy when AI predictions were wrong. The basic prediction produced the largest numerical top-one improvement, while the LLM explanation produced the smallest; however, the difference between those two explanation groups was not statistically significant. The evidence therefore does not show that elaborate explanations improved PCP accuracy more than simply displaying the prediction and confidence.

LLM explanations affected PCPs in another way. The correlation between confidence and accuracy increased from \(r=0.084\) without AI to \(r=0.494\) with LLM explanations. In other words, the narratives did not yield the largest accuracy gain, but they helped PCP confidence better reflect actual performance.

The medical-student comparison further suggests that expertise matters. On the same differential-diagnosis task, students were more likely than PCPs to follow the AI regardless of its correctness. The authors describe clinical knowledge as a cognitive firewall: a lay user may use an explanation to form a belief, while an experienced clinician can compare its specific claims with an existing diagnostic hypothesis.

That interpretation is plausible, but expertise should not be treated as complete protection. PCPs were not dermatologists, their initial task accuracy was low, and the study included only a small number of incorrect AI cases per person. Clinical experience reduced susceptibility in this setting; it did not make automation bias impossible.

The users who needed help were also easier to mislead

The authors classified a participant as fully deferential when their correct final answers occurred only on cases where the AI was correct and they remained incorrect whenever the AI was wrong.

In the general-public Human-First group, deferential users began with 65.8% accuracy, compared with 72.5% among non-deferential users. Deferential PCPs also had substantially lower initial performance. This creates a difficult design problem: the people who could gain the most from decision support may also have the least independent evidence with which to reject a convincing error.

This relationship is associative rather than causal. Deference was defined from participants’ response patterns after seeing only 12 cases, including approximately two AI errors, and low initial performance contributes mechanically to some of those patterns. Still, the broader implication is useful. A safe system cannot assume that every user has the same ability to audit an explanation.

Decision order changed anchoring more than final accuracy

By the final decision round, Human-First and AI-First workflows produced similar accuracy for both lay users and PCPs. However, showing AI first generally increased the proportion of users whose answers tracked the AI.

Among the general public, AI-First increased deference by an average of 8.4 percentage points across explanation types, although the covariate-adjusted comparisons were not statistically significant. Among PCPs, the largest descriptive increase occurred with LLM explanations at 19 points. These subgroup sizes were small, so the precise values should be interpreted cautiously, but the pattern is consistent with anchoring: once the AI supplies a diagnosis, subsequent reasoning begins from that suggestion.

The authors recommend a Human-First workflow. Even when final accuracy is unchanged, requiring an independent initial judgment preserves information that would otherwise be lost. A disagreement between the human and model can then trigger deeper evidence, a confidence check, or a second opinion rather than allowing the AI to silently define the starting point.

Limitations

Several design choices limit how far these results can be generalized:

• Each participant saw only 12 images in an online experiment without patient history, demographics, symptoms, longitudinal information, or laboratory data. This is much simpler than real dermatological diagnosis.
• AI accuracy was intentionally fixed near 80%, diseases and skin tones were balanced, and high-quality images were selected. Real deployment includes prevalence shift, clustered failures, uncertain image quality, and users who do not know how often the system is wrong.
• The LLM result is specific to a single-pass GPT-4V rationale generated from a fixed classifier label and a particular prompt. It does not evaluate end-to-end LLM diagnosis or prove that every natural-language explanation will have the same effect.
• The four XAI conditions communicate different amounts and types of information. Their effects combine explanation content, visual design, cognitive effort, and presentation style.
• In Study 2, the AI classified 34 diseases, while PCPs entered unrestricted free text aided by autocomplete containing 445 disease names. The study was designed to measure reactions to AI rather than compare human and model capability, but this asymmetry complicates interpretation of the very large accuracy improvement.
• Only 16.8% of the complete training image collection had expert-annotated Fitzpatrick labels; weaker image-derived skin-tone labels were used for fairness training on the remainder. The user-study images themselves used expert annotations and were balanced, but the light/dark grouping is still a coarse representation of skin appearance.
• Dermatologists’ ratings of explanation correctness and informativeness had only fair-to-moderate interrater reliability (\(\kappa=0.42\) and \(0.37\)), illustrating that explanation quality is itself difficult to define.

The revised analyses use linear mixed models with participant- and image-level effects and relevant covariates, which is stronger than treating every response as independent. Nevertheless, the number of subgroup and post-hoc comparisons means that marginal findings should be interpreted more cautiously than the main correct-versus-incorrect AI effects.

Thoughts

This paper changes how I think about the “black box” problem. Explainability is often presented as a model property: either the model can explain itself or it cannot. In practice, an explanation is part of an interaction among a model, an interface, a user, and a clinical decision. Its safety depends on the user’s expertise, the model’s correctness, and when the explanation is shown.

The same issue applies beyond dermatology. In cardiology, a model might highlight an ECG segment, retrieve similar echocardiograms, or generate a fluent rationale for a risk prediction. A visually or linguistically convincing explanation could help a clinician identify relevant evidence. It could also encourage the clinician to reinterpret ambiguous findings in support of an incorrect prediction.

For this reason, XAI evaluation should not stop at asking whether an explanation looks reasonable. We should measure the combined human-AI system separately when the model is correct and when it is wrong, across users with different levels of expertise, and under realistic decision workflows. A useful explanation should help a person verify or falsify a prediction—not simply make the prediction easier to believe.

The paper’s most actionable lesson may therefore be about interface design rather than explanation algorithms. Collect an independent judgment before revealing the AI. Display uncertainty and fallibility. Treat disagreement as a reason to inspect more evidence. Adapt the amount and type of explanation to the user. Most importantly, remember that a persuasive explanation can increase trust faster than it increases understanding.

Figures are reproduced without modification from Xu et al. for noncommercial use under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.