Deep Learning Combines ECG and Video to Predict Emergency Admissions

Researchers at National Taiwan University Hospital have demonstrated that a multimodal deep learning architecture combining standard twelve-lead electrocardiogram images and short respiratory video recordings can effectively predict hospital admissions directly from emergency department triage. Published in BioData Mining, the prospective investigation led by Guang-Yuan Chen, Li-Chen Fu, Chu-Lin Tsai, and colleagues evaluated how non-invasive physiological signals gathered during initial presentation could help clinicians identify patients requiring inpatient hospitalization. In an adult emergency cohort of 118 individuals, the primary cardio-respiratory fusion model attained an overall diagnostic accuracy of 0.814, alongside an area under the receiver operating characteristic curve of 0.863 and an area under the precision-recall curve of 0.675.

Emergency department overcrowding has become an increasingly urgent global public health crisis that compromises timely medical delivery and strains healthcare facility capacities. Excessive congestion within triage bays and acute treatment areas reduces hospital throughput, increases waiting times, forces frequent ambulance diversions, and elevates patient risks for adverse medical events, including extended inpatient stays and heightened mortality. Traditional triage frameworks, including the Emergency Severity Index and the Taiwan Triage and Acuity Scale, rely almost exclusively on manually recorded vital signs, chief complaints, and demographic profiles. However, these conventional scoring tools frequently struggle to predict ultimate patient disposition with consistent accuracy, resulting in misallocated treatment spaces and prolonged boarding times.

To establish an objective, automated assessment pipeline, the research team conducted a prospective observational study between February and December 2024 within the emergency department of National Taiwan University Hospital, a tertiary academic medical facility in Taiwan maintaining 2,400 inpatient beds and handling roughly 100,000 emergency patient visits annually. Dedicated research staff consecutively screened presenting individuals according to a standardized protocol, enrolling adults aged eighteen years or older capable of granting informed consent. The investigators explicitly excluded patients who required immediate cardiopulmonary resuscitation, individuals who were unconscious without an authorized legal representative present, patients exhibiting severe psychiatric disturbances impeding communication, and individuals diagnosed with active infectious illnesses requiring specialized isolation precautions. Ultimately, 118 eligible adult participants enrolled in the primary cohort, in which the observed baseline hospital admission rate was 27.1 percent, representing thirty-two admitted patients and eighty-six discharged patients.

Data collection during triage commenced under standard ambient lighting conditions, where investigators positioned an Apple iPhone 13 on a stabilized mount to capture a standardized thirty-second video recording of each enrolled patient. To safeguard personal privacy while capturing breathing movements, the camera recorded the clothed chest wall, neck, and lower facial region covered by a protective face mask, maintaining front-facing orientations within fifty degrees. The image analysis pipeline first utilized an adapted YOLOv5 object detector, fine-tuned on an annotated mask dataset to achieve a recall of 0.90 and a mean average precision of 0.95, to locate the face mask and isolate a rectangular chest region of interest directly below. This cropped video sequence was processed through a three-dimensional convolutional neural network encoder-decoder architecture coupled with a bi-directional long short-term memory network. The resulting estimated waveform was refined using a physiological bandpass filter restricted between 3 and 60 breaths per minute, demonstrating a 0.73 Pearson correlation against reference respiratory rates annotated independently by board-certified emergency physicians, before conversion into standardized high-contrast respiratory curve images.

In parallel with respiratory imaging, the clinical framework incorporated standard twelve-lead electrocardiograms whenever triage nurses or protocol guidelines indicated suspicion of acute coronary syndromes, cardiac arrhythmias, unexplained syncope, or critical medical instability. These clinical recordings were retrieved directly from portable document format reports, which frequently presented visual artifacts, diagnostic text labels, lead demarcations, and background measurement grids that could introduce algorithmic confounding during deep learning feature extraction. The researchers deployed a specialized visual preprocessing pipeline designed to eliminate extraneous background data while preserving critical cardiac voltage waveforms. Initial thresholding applied to the red color channel effectively suppressed background grid matrices, after which automated contour detection isolated and removed non-waveform visual labels and textual notations. Successive morphological opening using a three-by-three kernel smoothed boundary variations, inverted pixel values to depict white voltage traces across a uniform dark background, applied median filtering alongside histogram equalization to maximize local contrast, and finalized high-contrast binary electrocardiographic representations suitable for deep convolutional encoding.

For feature representation, investigators implemented a ConvNeXt-T convolutional architecture as the shared neural backbone for both visual streams, selecting the model for its computational efficiency and parameter economy on modest clinical sample sizes. To enable the network to prioritize meaningful structural signals over irrelevant background regions, lightweight spatial attention modules derived from the Convolutional Block Attention Module architecture were positioned after stages one through four of each ConvNeXt-T trunk. These attention blocks applied channel-wise average and max pooling, combined the resulting descriptors into a unified two-channel representation, executed a spatial convolution with sigmoid activation, and multiplied the derived attention mask element-wise against intermediate feature maps. Because electrocardiograms were captured in only a fraction of triage visits, training an electrocardiographic classifier solely from scratch presented substantial overfitting risks. To overcome this limitation, the authors pretrained the electrocardiogram branch on an external collection of 713 emergency electrocardiogram records sourced from Far Eastern Memorial Hospital, a separate tertiary institution managing 130,000 annual emergency visits, before transferring the tuned weights into the multimodal pipeline.

To accommodate operational scenarios where emergency patients do not receive an electrocardiogram, the authors engineered an indicator-gated dual-branch classifier that dynamically adapts to missing diagnostic modalities without requiring artificial imputation. A binary indicator variable tracked whether electrocardiographic data were present for an individual encounter. When electrocardiograms were available, the architecture concatenated the spatial electrocardiogram feature vector, the extracted respiratory feature vector, and the binary flag into a comprehensive multi-layer perceptron classifier branch. Conversely, when electrocardiographic testing was absent, the network routed only the respiratory feature embedding and the status indicator through an alternate single-modality classification branch, enabling graceful operational fallback. Furthermore, because discharged presentations substantially outnumbered hospitalizations in the development cohort by a ratio of approximately 2.7 to 1, the authors utilized a class-balanced loss formulation. Built upon binary cross-entropy with unnormalized logits, this loss function adjusted individual training sample contributions through an effective sample size scaling parameter, preventing algorithmic optimization from collapsing into majority-class prediction while elevating diagnostic sensitivity for admitted patients.

Model evaluation across the 118-patient primary cohort followed a stratified four-fold cross-validation scheme structured to evaluate fold-wise averages across balanced admission frequencies. On this primary dataset, the indicator-gated multimodal fusion model achieved an average diagnostic accuracy of 0.814, a sensitivity recall of 0.749, a precision of 0.635, an F1-score of 0.680, an area under the receiver operating characteristic curve of 0.863, and an area under the precision-recall curve of 0.675. Across four independent random split seeds, the model maintained consistent discrimination, yielding receiver operating characteristic areas spanning 0.843 to 0.863. In comparative testing against standard practice, the multimodal framework noticeably surpassed an enhanced baseline logistic regression model parameterized with Taiwan Triage and Acuity Scale classifications, patient age, gender, heart rate, and manually counted respiratory rate. While an isolated respiration-only branch produced an area under the receiver operating characteristic curve of 0.810, the joint fusion system demonstrated superior predictive discrimination. An exploratory configuration combining deep cardio-respiratory embeddings with structured triage variables achieved an area under the receiver operating characteristic curve of 0.893.

Targeted subgroup investigations and algorithmic ablation experiments further clarified the operational contributions of specific architectural mechanisms within the dual-branch framework. Among a subgroup of thirty-three patients presenting with both electrocardiograms and video recordings, the integrated fusion model boosted sensitivity recall from 0.611 in single-modality analysis up to 0.944, reflecting the higher underlying hospitalization rate of patients selected by triage nurses for cardiac testing. Calibration analysis utilizing five probability bins across out-of-fold predictions demonstrated strong diagnostic reliability, producing a Brier score of 0.161 alongside an expected calibration error of 0.094. Component ablations confirmed that spatial attention mechanisms and class-balanced loss functions each generated independent performance enhancements, whereas eliminating external transfer pretraining or removing the binary gating indicator precipitated notable degradations in discriminative accuracy. To investigate feature interpretability, the team generated Score-Class Activation Mapping visualizations across respiratory waveform curves, revealing that neural attention systematically centered upon signal inflection peaks and trough turning points, which correspond to physiological respiratory amplitude excursion and breathing frequency.

To examine temporal consistency within the same medical institution, the researchers evaluated the trained multimodal network against an independent prospective holdout cohort consisting of twenty consecutive emergency patients enrolled between January and March 2026. Without model retraining, the fusion classifier sustained an area under the receiver operating characteristic curve of 0.780, an area under the precision-recall curve of 0.811, and an overall classification accuracy of 0.750. Nevertheless, the authors emphasized multiple methodological boundaries governing their findings, noting that the single-center dataset was modest in size, that patient cohorts excluded acute resuscitation cases and isolated infectious patients, and that model performance was verified offline rather than embedded within real-time clinical workflows. Additionally, institutional admission choices naturally reflect physician practices, bed availability constraints, and social support factors rather than pure biological severity alone. Consequently, the authors concluded that while non-invasive cardio-respiratory video and image analysis represents a promising technical foundation for automated triage support, extensive multi-center clinical validations, real-time workflow trials, and comprehensive privacy protections remain mandatory before formal bedside implementation.

Subject of Research: Multimodal deep learning for emergency department hospital admission prediction

Article Title: Cardio-respiratory deep learning model to predict hospital admission from emergency department electrocardiogram images and respiration videos: a prospective study

Article References: Chen, G.-Y., Lu, S.-C., Cheng, W.-H., Gao, J.-W., Sun, J.-T., Huang, C.-H., Tsai, C.-L., & Fu, L.-C. (2026). Cardio-respiratory deep learning model to predict hospital admission from emergency department electrocardiogram images and respiration videos: a prospective study. BioData Mining, 19(1), Article 77. https://doi.org/10.1186/s13040-026-00583-9

Image Credits: AI Generated

DOI: 10.1186/s13040-026-00583-9

Keywords: Emergency Medicine, Deep Learning, Electrocardiography, Computer Vision, Triage Prediction, Hospital Admission, Multimodal AI, Physiological Monitoring, Cardio-respiratory, deep, learning, model