The Hidden Bias in AI Healthcare May Come From Clinicians, Not Algorithms

Artificial intelligence has swept into hospitals and clinics with a bold promise: smarter diagnoses, faster decisions, and more efficient care. In response to growing evidence that algorithms can perpetuate health inequities, developers have poured enormous effort into technical fixes—diversifying training datasets, running fairness audits, and building more interpretable models. But a provocative new perspective published in the Journal of General Internal Medicine argues that the industry has been fixing the wrong end of the problem. Even a mathematically fair algorithm, the authors contend, can produce deeply unfair outcomes once it passes through the hands of the clinicians who use it.

The perspective, written by Kellie Owens, Maggie Ramatowski, and Saul B. Blecker of New York University Grossman School of Medicine, draws on social science research and the authors’ own experiences deploying AI models in real healthcare settings. Their central claim is technically precise and socially explosive at the same time: implementation itself is a major driver of bias in healthcare AI. The ways AI systems are adopted, interpreted, and applied in practice—shaped by institutional norms, professional hierarchies, and patient characteristics—can reproduce or even amplify inequities that no dataset diversification could ever prevent.

The technical foundation of the argument rests on a distinction that fairness engineering has largely ignored. Most current approaches to algorithmic fairness draw on principles of distributive justice, asking whether a model allocates recommendations or resources equitably across socially salient groups such as race, gender, or age. A model might show similar sensitivity or positive predictive value across demographic groups—hallmark mathematical signals of fairness—and still introduce bias at the point of use. The reason is that these metrics presume model outputs directly drive decisions, when in reality clinicians serve as crucial intermediaries. Clinicians choose whether to consult a model at all, and each decision point introduces variation, and with variation comes the potential for bias.

Social scientists have long documented that algorithmic outputs are not inert facts but tools shaped by interpretation and context. Prior research on how clinicians act on algorithmic input shows that they may use or ignore recommendations based on their own judgments of the model’s value, and that healthcare workers and patients hold different expectations that shape how data is understood. AI outputs become embedded in complex feedback loops, altering individual decisions, broader workflows, and professional norms. The introduction of AI into clinical care is deeply social, woven into institutional routines, power dynamics, and historically rooted patterns of discrimination. Understanding these dynamics, the authors insist, requires looking beyond the algorithm itself.

Although the authors are unaware of studies directly examining bias in the implementation of AI-based algorithms in healthcare, emerging evidence from clinical decision support systems offers troubling clues. Studies indicate that the effectiveness of decision support tools can differ across gender and age, and some reports suggest providers may incorporate recommendations differently depending on patient characteristics. These patterns of uneven uptake may reflect familiar dynamics—workflow constraints, trust levels, or cognitive biases such as automation bias, in which clinicians defer to system outputs without adequate scrutiny. Clinicians often do not even know whether a decision support tool is AI-based.

Yet AI may introduce distinct challenges of its own. Because AI systems are often perceived as more objective or personalized than traditional decision support, they may heighten automation bias, making clinicians more likely to accept outputs at face value. Conversely, limited AI literacy or lack of transparency can make it difficult for clinicians to appropriately evaluate model outputs, producing overreliance in some cases and unwarranted skepticism in others. In the authors’ own research and experience, clinicians may use AI models differently depending on which patient is in front of them, mirroring existing power dynamics and social hierarchies. Clinicians may be more likely to accept AI recommendations at face value for marginalized patients, while subjecting outputs to greater scrutiny or repeated testing for more socially advantaged patients. Fairness, in other words, is entangled with dynamics of trust, authority, and power—not reducible to model accuracy alone.

This finding strikes at the heart of the most widely proposed safeguard in AI governance: the human-in-the-loop. The assumption behind this strategy is that clinicians, by exercising judgment, will act as a fail-safe against algorithmic error or bias. The authors suggest the opposite. Clinicians themselves are a major site where inequities emerge, and placing primary responsibility for fairness on individual clinicians risks reinforcing rather than mitigating structural inequities. Humans, they write, are often the very conduit through which inequities enter the system. Institutions that assume human oversight guarantees fairness are making a dangerous assumption—one that recent literature has begun to question, with some proposals favoring supervisory oversight over more autonomous systems instead of active clinician-AI collaboration.

To move from descriptive concern to testable science, the authors lay out a concrete research agenda. One approach is to examine whether and how clinicians differentially use, ignore, or override AI recommendations based on patient characteristics such as race, gender, or socioeconomic status, using audit log data linked to clinical and demographic information. A second line of inquiry should focus on conditions of use: whether time pressure, workload, or alert fatigue increases reliance on AI, and whether these effects vary across patient groups. Qualitative and ethnographic approaches—observational studies of clinical workflows and in-depth interviews with clinicians and patients—can surface how AI outputs are made meaningful in context and how competing definitions of fairness are reconciled in practice. Finally, linking patterns of AI use to downstream clinical decisions and outcomes is essential to determine whether these dynamics translate into inequities in care.

On the governance side, the authors call for a shift from human-in-the-loop to system-in-the-loop oversight, placing responsibility across healthcare organizations, regulators, and technology vendors. That framework includes continuous monitoring for disparities in AI use and outcomes across patient populations, with dashboards that track differential exposure to AI recommendations, override rates, and downstream outcomes by relevant subgroups. Institutional policies should standardize when AI recommendations must be considered or documented—for example, requiring clinicians to provide brief justification when overriding certain recommendations, or generating periodic audit reports for review by governance committees. Workflows should account for clinician workload and decision fatigue, recognizing that alert fatigue or lack of time may systematically disadvantage some groups of patients more than others. Vendors, too, should bear greater responsibility for post-deployment monitoring, transparency, and collaboration with health systems, including providing tools for subgroup performance auditing and tracking clinician interaction patterns.

The authors are careful to note that differences in how AI tools are used across patient groups are not necessarily evidence of unfair implementation. A high override rate for a particular group could indicate either appropriate clinical correction of model outputs or inequitable disregard of recommendations—disentangling the two is precisely what systematic tracking is for. But the broader message is unambiguous. Fairness is not only designed into models; it is negotiated and enacted in practice. By acknowledging that clinicians are not neutral backstops but active participants whose judgments are socially patterned, healthcare systems can design governance strategies that take human behavior seriously. The future of equitable AI in medicine, this perspective argues, depends less on perfecting the algorithm and more on governing the messy, human system around it.

Subject of Research: Health equity and AI governance in clinical decision support

Article Title: Promoting Fairness in AI Implementation

Article References: Owens, K., Ramatowski, M., & Blecker, S. B. (2026). Promoting Fairness in AI Implementation. Journal of General Internal Medicine. https://doi.org/10.1007/s11606-026-10767-y

Image Credits: AI Generated

DOI: 10.1007/s11606-026-10767-y

Keywords: artificial intelligence, healthcare AI, algorithmic bias, clinical decision support, health equity, human-in-the-loop, AI governance, medical ethics, Promoting, Fairness, Implementation, scientific research