Emergency Department Triage Algorithms Rarely Integrate User-Centred Interface Testing or Design

Emergency department triage represents a promising application for machine learning decision support, yet systems developed to assist clinical staff in prioritizing patients rarely incorporate structured interface design or iterative usability testing. A systematic review published in Frontiers in Digital Health by Kirsten Zantvoort and colleagues from Marburg University and University Hospital Giessen and Marburg reveals that despite extensive artificial intelligence research into clinical triage, predictive systems fail to integrate human-centered design into clinician displays. The researchers conducted an exhaustive examination of global literature, screening 2,678 records and identifying 25 primary studies that evaluated clinician-facing interfaces for machine learning decision support in routine emergency triage. Their findings demonstrate a striking disconnect between predictive modeling and acute care practice: while 80 percent of systems were published between 2022 and 2025, only 36 percent were integrated into clinical workflows, and virtually none reported structured usability evaluations or involved human-computer interaction experts during system development.

Emergency departments represent uniquely demanding, high-stimulus environments characterized by persistent time pressure, frequent workflow interruptions, ambient noise, and heavy cognitive loads. Triage clinicians must rapidly establish clinical urgency, where triage decisions can be lifesaving or life-ending and often leave little time to rectify errors if they occur. Although substantial research demonstrates the theoretical accuracy of predictive machine learning models in retrospective simulations, real-world clinical adoption remains strikingly limited across acute care settings. Suboptimal interface design is a well-established cause of user fatigue and subsequent disengagement. Ill-designed software can exacerbate workload, cause clinical stress, and induce automation bias or alert fatigue, prompting clinicians to disregard valuable guidance or make erroneous decisions. Human-computer interaction principles emphasize that placing user needs at the center of interface development is essential for realizing clinical benefit. However, whether these theoretical design ideals are translated into practice within emergency decision support tools has remained largely unexamined until this review.

To investigate interface characteristics and development methodologies across triage tools, the authors conducted their systematic review in accordance with PRISMA 2020 guidelines under pre-registered PROSPERO protocol CRD420251119077. They searched PubMed, Embase, CENTRAL, Web of Science, and IEEE Xplore for studies published between 2015 and August 19, 2025. Demonstrating methodological transparency, the authors documented a technical challenge during screening: although they initially applied the machine learning active-learning package ASReview to prioritize records, the algorithm overfitted to clinical keywords and failed to identify the atypical criterion of interface presence, missing relevant studies across iterations. Consequently, the team abandoned automated prioritization and manually screened all 2,678 records with two independent researchers, achieving an interrater reliability of 0.98. Full texts of 57 candidate papers—including 55 from database queries and two from citation screening—were assessed, ultimately identifying 25 eligible primary studies evaluating machine learning decision support for routine emergency department triage.

The 25 included studies displayed broad geographic distribution alongside notable concentration in recent years, with 80 percent published between January 2022 and August 2025 and only two prior to 2019. Regionally, ten studies were conducted in Asia, seven in Europe, six in North America, one in Australia, and one in Africa. Despite this research acceleration, the developmental maturity of examined systems was predominantly preliminary. Only four papers (16 percent) described tools fully deployed within routine emergency department patient care. Conversely, one paper (4 percent) presented an early concept proposal, while a plurality of 12 papers (48 percent) described technical prototypes or feasibility analyses without any user involvement. Three studies (12 percent) evaluated user interactions solely in laboratory research settings utilizing historical records or artificial vignettes without real patients. The remaining five studies (20 percent) conducted clinical tests within research contexts without routine care integration. This distribution indicates that most interface designs reflect early developmental phases rather than battle-tested clinical implementations.

Regarding decision control, 88 percent of reviewed systems (22 studies) were structured to maintain clinicians in full decision control, showing outputs but requiring humans to enter final decisions. Two studies utilized active approval models where clinicians actively accepted or rejected predictions, while only one system implemented automated decisions. That deployment, described by Goertzen and colleagues, utilized automated prioritization to address an absence of standardized triage in a resource-limited setting rather than replacing clinical staff. No system employed reactive approval, where automated decisions proceed unless vetoed by staff. In clinical scope, 14 systems (56 percent) addressed general triage acuity assignment, nine of which implied specific clinical recommendations. Ten studies (40 percent) addressed sub-problem triage, focusing primarily on sepsis risk or clinical deterioration; eight presented raw numeric risk scores or binary flags representing low actionability. Only one study addressed next-step identification, predicting requirements for six critical emergency interventions on continuous scales and binary recommendations.

System display formats varied considerably across text, visual cues, and numeric data, though no system innately incorporated audio alerts. Text was the most prevalent representation format, utilized in 17 systems (68 percent), followed by visual components in 16 systems (64 percent) and numeric values in 13 systems (52 percent). Notably, numeric outputs were never presented alone: five systems utilized text alone, while three used visual cues alone. The authors categorized eight systems (32 percent) as single-format, 13 (52 percent) as dual-format, and four (16 percent) as triple-format combining all modalities. Among single-format systems, six operated through pull-based mechanisms, largely because five were stand-alone web applications separate from primary records. Conversely, triple-format systems frequently featured two-step displays. For instance, Chang and colleagues presented an initial graphical warning icon on patient overviews that revealed five risk scores, categorical interpretations, and contributing variables when clicked. Similarly, the KATE system pushed visual alerts only when recommendations deviated from triage nurses’ assessments.

The systematic review highlighted pronounced gaps in displaying model metadata, predictive performance, and statistical uncertainty. While explainability features were relatively common—appearing in nine systems, including seven dual-format tools that displayed contributing clinical parameters—authors emphasized that local algorithmic explanations can be mathematically valid yet clinically implausible. Crucially, systems rarely communicated their operational validation performance or prediction uncertainty. Only three dual-format systems (23 percent) presented performance metrics such as receiver operating characteristic curves, and only one system incorporated confidence intervals to convey statistical uncertainty. In single-format systems, neither metadata nor uncertainty indicators were ever reported.

Developmental methodologies and user-centered design processes were seldom documented among the analyzed literature. Across all 25 studies, adherence to an established design framework was reported only once: Williams and colleagues explicitly aligned and evaluated their system using the Five Rights of Clinical Decision Support framework. Four papers mentioned interface ease of use, but none defined usability criteria or provided empirical evidence of user friendliness. Furthermore, interdisciplinary collaboration revealed critical domain deficits. While clinical researchers were present in 22 author lists (including emergency specialists in 14) and computational researchers in 11, not a single paper reported the involvement of human-computer interaction or user experience specialists. Structured usability testing was similarly rare; Williams and colleagues combined observations, interviews, and formal usability testing, while Goertzen and colleagues utilized iterative stakeholder feedback. Most other studies acknowledged the absence of usability evaluations or examined acceptability informally post-deployment, while formal training protocols were documented in only two self-developed systems.

The review identified substantial differences between commercial systems and academic, self-developed solutions across interface design and clinical implementation. Commercial tools comprised 36 percent of the sample (nine studies), including two evaluations based on the TriageGO platform. Commercial solutions exhibited more elaborate displays than self-developed tools: 78 percent featured dual-format or triple-format interfaces, compared to 63 percent for self-developed systems, while both single-format commercial systems were general-purpose large language models. Furthermore, only half of identified commercial products were purpose-built for emergency department environments, frequently originating from general practice or differential diagnosis applications. Interestingly, while studies examining commercial tools reported user-centered evaluations more frequently (44 percent versus 19 percent), only one commercial paper described underlying tool development, and none discussed user training protocols.

Synthesizing their descriptive findings, the systematic review authors emphasized that none of the 25 included studies empirically tested whether specific user interface design choices directly influenced clinical decision quality, diagnostic accuracy, or patient outcomes. The team noted key study limitations, including pronounced heterogeneity across prediction tasks and settings that precluded meta-analytic pooling, reliance on published or obtained reports that may omit unstated design steps, and restricting scope to clinician-facing emergency department tools. To address the persistent implementation gap separating machine learning potential from emergency department reality, the investigators called for rigorous adherence to the DECIDE-AI reporting guideline. They specifically urged authors, reviewers, and journal editors to enforce comprehensive reporting of interface screenshots, interaction modalities, human factor frameworks, usability evaluations, and clinical learning curves. The researchers concluded that establishing transparent, interdisciplinary interface design standards remains essential for determining whether user-centric improvements can ultimately translate algorithmic accuracy into tangible benefits for clinical triage and emergency patient care.

Subject of Research: User interface design and testing in machine learning triage systems

Article Title: User interfaces in machine learning-based decision support for emergency department triage — a systematic review

Article References: Zantvoort, K., Brouwer, S., Schlicker, N., Mross, P., Jerrentrup, A., Hirsch, M. C., & Russ, P. (2026). User interfaces in machine learning-based decision support for emergency department triage — a systematic review. Frontiers in Digital Health, 8, 2004648. https://doi.org/10.3389/fdgth.2026.2004648

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.2004648

Keywords: emergency department triage, clinical decision support systems, machine learning, user interface design, human-computer interaction, systematic review, usability testing, electronic health records, DECIDE-AI, PROSPERO, PRISMA 2020, automation bias, explainability, predictive uncertainty, sub-problem triage, acuity assessment, workflow integration, clinical adoption