In an era when artificial intelligence systems are increasingly entrusted with decisions that carry life-and-death consequences, one of the most persistent anxieties surrounding their use in medicine has been the so-called black box problem: the inability of clinicians to see inside these systems and understand why a particular prediction or diagnosis was produced. Now, researchers at King’s College London, working in collaboration with the Alan Turing Institute and supported by the Turing-Roche Strategic Partnership, have taken a significant step toward solving this problem, not by building a new type of AI, but by developing a rigorous mathematical framework that can determine whether an AI system that appears to explain its own decisions is genuinely transparent, or merely pretending to be.
The research, published in the Journal of Machine Learning Research, addresses a subtle but critically important deception that can occur within a family of AI models known as concept-based models. These systems were designed specifically with human oversight in mind. Rather than processing raw data, such as the pixel values of a medical scan, and emitting an inscrutable answer, concept-based models are constructed to reason through clinically meaningful intermediate variables: blood pressure readings, the presence or absence of fever, the size of a tumour, or the visibility of abnormalities in a medical image. The promise is seductive and intuitive. If a model must justify its conclusions in terms of concepts that a trained clinician can inspect, verify, and contest, then the clinician retains meaningful control over the decision-making process. The AI becomes less of an oracle and more of a colleague whose reasoning can be followed.
In principle, this is exactly what emerging AI regulations around the world demand. Regulators increasingly emphasise transparency and meaningful human oversight as prerequisites for deploying automated systems in high-stakes environments, and healthcare is the most consequential of these. Clinicians need to be able to identify when a system has made a mistake and to intervene in its decision-making before harm occurs. An AI that predicts patient outcomes or flags signs of disease with high accuracy is of limited value, and potentially considerable danger, if the experts responsible for patient care cannot audit the chain of reasoning that led to its conclusions.
But here lies the problem the King’s College team set out to expose. Concept-based models, for all their apparent clarity, can suffer from a phenomenon known as information leakage. This occurs when the concepts used by the model contain additional, unintended information that is not visible to the human reviewing the decision. The concept label might say “fever present” or “tumour size: moderate,” and the human reviewer sees exactly that. But hidden within the numerical representation of that concept may be a wealth of auxiliary information, correlations and signals inherited from the raw data, that the model exploits when forming its final prediction. The consequence is profound: the model appears interpretable while still relying on information that a human cannot see or assess. In effect, it behaves like a black box model disguised as an interpretable one. The transparency is an illusion, and the clinician’s sense of oversight is misplaced confidence rather than genuine understanding.
The scale of this problem becomes clear when one considers what is at stake. If a concept-based diagnostic system recommends an aggressive treatment pathway, and the apparent basis for that recommendation is a set of concepts the clinician has reviewed and found reasonable, the clinician may approve the recommendation believing they have verified its justification. If, however, the model’s actual decision was driven substantially by hidden information smuggled inside those concepts, the clinician has not verified anything of consequence. The audit has failed silently, and the regulatory requirement for meaningful human oversight has been satisfied only in appearance.
Distinguishing genuine transparency from its counterfeit has, until now, lacked a rigorous foundation. It is one thing to suspect that leakage might occur; it is quite another to define it precisely enough to measure it. This is the central contribution of Dr Enrico Parisini, Senior Research Fellow in Machine Learning and first author of the paper, and Dr Chris Banerji, AI+ Senior Fellow and senior author, who developed a mathematical framework that defines and quantifies information leakage in concept-based AI systems with exactitude. The framework consists of two complementary measures. The first, concepts-task leakage, abbreviated CTL, captures hidden information linked to the final prediction, that is, the extent to which the model’s output depends on information concealed within the concepts rather than on the concepts as the human perceives them. The second, interconcept leakage, or ICL, captures hidden information shared between concepts, revealing how the internal representations of different concepts may be entangled in ways invisible to the reviewer.
Together, these two measures provide a diagnostic instrument of genuine practical power. The researchers tested their framework across several datasets and found that it could reliably detect leakage when it was present. More impressively, the framework could predict how models would respond when their concepts were deliberately changed. This is a crucial validation: a measure of leakage is only useful if it reflects real causal structure within the model, and the ability to anticipate model behaviour under intervention demonstrates that the framework captures something true about the system’s internal workings, not merely a statistical artefact.
The implications reach well beyond diagnostics of existing systems. Because the framework quantifies leakage rather than merely detecting it, it provides practical guidance for designing concept-based models that minimise leakage from the outset. Developers can, in effect, use the measures as design targets, iteratively refining their architectures until the concepts presented to the human user genuinely carry the information that drives the decision. This transforms transparency from a vague aspiration into an engineering specification with a measurable standard, something that has been conspicuously absent from the discourse around explainable AI.
The work also arrives at a moment of acute relevance for the regulation of medical AI. As health systems around the world begin to deploy machine learning tools for triage, diagnosis, and prognostication, regulators are grappling with how to certify that these systems are safe and auditable. A framework that can quantify whether a model’s claimed interpretability is genuine offers regulators and developers a common language for assessing transparency claims. It moves the conversation from reassurance to measurement, from assurances that a model “uses clinically meaningful concepts” to a demonstrable quantity describing exactly how much hidden information those concepts conceal.
The researchers themselves frame the contribution as a deliberate act of caution in a field defined by speed. Dr Banerji, the senior author, described the motivation bluntly: rather than putting the cart before the horse, while the field of AI is moving quickly to apply models to real-world problems, the team stepped back to consider what is needed to make these systems safe and reliable. The focus, he explained, has been on understanding limitations and addressing them, so that the deployment of these models in clinical practice can proceed in a safer way. It is a philosophy that stands in quiet contrast to the prevailing urgency of the AI race, and one that may prove prescient as the first generation of clinical AI deployments encounters the messy realities of medical practice.
Dr Parisini, the paper’s first author, emphasised the dual nature of the finding. Concept-based AI has genuine potential to make AI systems more transparent, he noted, but the research shows that models can appear interpretable while still relying on information hidden from the person using them. By identifying and measuring this hidden information, he argued, the field can take concrete steps toward AI systems that are transparent in substance rather than in appearance. It is a distinction that may come to define the next phase of the explainable AI movement: the difference between systems that look accountable and systems that are accountable.
The team is now working to apply these approaches to real clinical problems, a step that will test whether the framework’s mathematical guarantees translate into safer deployments in hospitals and clinics. If they do, the work could become a standard component of the AI development pipeline, a verification layer through which every concept-based clinical model must pass before it is trusted with patients. The research was supported by the Turing-Roche Strategic Partnership, the King’s College London AI+ Fellowship, and PharosAI.
For a field that has long promised transparency as the antidote to the black box, this study delivers an uncomfortable but necessary message: interpretability itself can be faked, sometimes without anyone intending it, and only mathematics can tell the difference. In making that difference measurable, the King’s College London researchers have given clinicians, regulators, and developers something they have lacked until now, a way to know whether the window they are looking through into an AI’s mind is truly open, or merely a mirror.
Subject of Research: A mathematical framework for defining and measuring information leakage in concept-based AI systems, to determine whether such systems are genuinely transparent or conceal hidden information from human reviewers.
Subject of Research: Medicine
Article Title: Mathematical framework could improve transparency of AI in clinical settings
Article References: Journal of Machine Learning Research, Vol. 27 Original publication
Image Credits: AI Generated
DOI: Not provided
Keywords: concept-based AI, information leakage, explainable AI, black box problem, clinical AI, human oversight, transparency, machine learning, healthcare AI, Journal of Machine Learning Research, King’s College London, medical decision-making
Cite Scienmag News
APA MLA Chicago
Reid Dalton. (September 9, 2026). New mathematical framework aims to make clinical AI more transparent. Scienmag. https://scienmag.com/new-mathematical-framework-aims-to-make-clinical-ai-more-transparent/
Reid Dalton. “New mathematical framework aims to make clinical AI more transparent.” Scienmag, 9 September 2026, https://scienmag.com/new-mathematical-framework-aims-to-make-clinical-ai-more-transparent/. Accessed 9 September 2026.
Reid Dalton. “New mathematical framework aims to make clinical AI more transparent.” Scienmag. September 9, 2026. https://scienmag.com/new-mathematical-framework-aims-to-make-clinical-ai-more-transparent/
Copy citation Download RIS
Tags: advanced techniques for AI decision understandingAI black box problem in medicineAI decision interpretability in healthcareAI model transparency validationAI model validation and verificationAI trustworthiness in clinical decision-makingAI with human oversight in healthcareblack box problem in medicineclinical AI accountabilityclinical AI decision transparencyconcept-based AI modelsconcept-based models in healthcaredevelopment of transparent AI in healthcareexplainable artificial intelligence in clinical diagnosisexplainable artificial intelligence in medicinehuman oversight in medical AI systemsmathematical framework for AI interpretabilitymathematical framework for AI transparencymeasuring AI explainability in medicinemedical diagnosis explainabilitytransparency assessment in AI systemstransparent AI in medicinetransparent clinical AItrust and accountability in clinical AI
