The Black Box Problem in Medical AI
Addressing interpretability, trust, and explainability in deep learning models for clinical pathology — and why EEHLSS’s HemaPath engine is designed to show its work.
When AI Knows but Cannot Explain
Deep learning models can classify haematological malignancies with accuracy approaching expert haematopathologists. This is one of the most important developments in diagnostic medicine in decades. It is also, currently, one of the most dangerous — because accuracy without interpretability is not clinically deployable medicine.
The term “black box” refers to a model which produces outputs without any accompanying explanation of which input features drove those outputs. Modern deep learning architectures are black boxes by nature — they learn representations distributed across billions of parameter values, not localised to any interpretable feature that a human can examine.
A medical AI that classifies a bone marrow aspirate cell as a blast with 87% confidence, and provides no further explanation, places an unacceptable burden on the clinician who must decide whether to trust the output. The black box problem in medical AI is not primarily a technical problem. It is a clinical governance problem, a regulatory problem, and a trust problem.
Three Reasons Opacity Fails Medicine
Reason 1: Clinical Accountability
When a haematopathologist classifies bone marrow blasts as AML, they are accountable — they can explain the morphological features: chromatin pattern, nucleolar prominence, N/C ratio. When an AI model produces the same classification with 92% confidence, the clinician who relies on that output is accountable — but has been given nothing to evaluate the AI’s reasoning against.
Reason 2: Error Detection and Failure Modes
Black-box models fail silently. Interpretable AI fails differently: when an attention map highlights the wrong region of a cell image, a reviewing expert can detect the error. Explainability does not guarantee accuracy — but it enables auditability.
Reason 3: Regulatory and Ethical Requirements
The EU AI Act mandates transparency requirements incompatible with pure black-box clinical AI. The FDA’s emerging framework for AI/ML-based Software as a Medical Device similarly emphasises performance monitoring and interpretability.
“An AI model that cannot explain its decision is not a diagnostic tool. It is an oracle — and medicine abandoned oracles when it adopted evidence-based practice.”— EEHLSS Computational Haematology Team
Explainable AI: Methods and Trade-offs
Attention Maps & GradCAM
Vision transformers can produce spatial attention maps showing which regions of an input image most influenced the classification. In haematological morphology, these maps should highlight the nucleus, nucleolus, chromatin pattern, and cytoplasm — the features a haematologist examines.
SHapley Additive Explanations
SHAP quantifies the contribution of each input feature to a model’s output. A SHAP explanation for an AML risk model might show that blast percentage contributed +0.35, WBC count +0.18, and haemoglobin −0.05 to a classification probability.
LIME
Local Interpretable Model-agnostic Explanations approximates a complex model locally with a simple interpretable model. Useful for text and tabular explanations; less reliable for high-resolution pathology images.
Concept Bottleneck Models
CBMs explicitly map inputs to human-interpretable concepts (e.g., “nuclear chromatin is fine”, “nucleolus is prominent”) before making a final classification, providing the closest analogy to how a haematopathologist actually reasons.
HemaPath: Designed to Show Its Work
The HemaPath morphology engine in MedLabAI-LIS is designed from the ground up to be interpretable. Every cell classification produced by DinoBloom-L is accompanied by three interpretability outputs: a confidence score, an attention map, and a structured feature attribution list.
Confidence Scores and Calibration
HemaPath applies temperature scaling calibration to convert raw softmax outputs to calibrated probability estimates. Any classification with calibrated confidence below 0.75 is automatically flagged for expert review.
Attention Map Integration
DinoBloom-L natively produces multi-head attention maps visualised as colour-coded overlays on the original cell image. An expert reviewing a HemaPath blast classification can verify that the attention map highlights nuclear features that define blast morphology.
Feature Attribution in Reports
HemaPath produces structured feature attribution lists for each classification, incorporated into the LLaMA 3.1 8B report narrative — producing haematopathology reports that explain the morphological reasoning behind each classification.
All HemaPath-generated reports are draft reports pending mandatory review and authorisation by a licensed medical laboratory scientist or haematopathologist. This satisfies ISO 15189:2022 requirements for result validation.
Leave a Reply