Surgical AI on a Clock: New Review Maps How Real-Time Operating Room Models Are Built

Real-time artificial intelligence for the operating room has moved far beyond lab curiosities, and a narrative review published in BMC Artificial Intelligence maps exactly how the field got there. The review, led by Naivedhya Agrawal and Minal Kharat, synthesizes 78 sources drawn from PubMed/MEDLINE, IEEE Xplore, Scopus, Web of Science, Google Scholar and computer-science repositories including arXiv and conference proceedings from MICCAI, NeurIPS, CVPR and ICRA, covering literature from January 2015 to April 2026. Its focus is the architecture stack behind systems that watch surgical video and identify what is happening during an operation: temporal modeling of frame sequences, fusion of heterogeneous data streams, and the compression tricks that let heavy networks run on edge hardware inside the theater.

The core technical problem is temporal. Surgical procedures last hours, follow structured phase sequences, and yet individual phase durations vary substantially with patient anatomy, pathology and surgeon technique. Recurrent neural networks, particularly LSTMs, established feasibility: SV-RCNet paired a ResNet for spatial features with an LSTM for temporal modeling, and cascaded two-LSTM designs learned dependencies first from short clips and then across the full video. But recurrence carries a structural penalty. Because sequential dependencies prevent parallel computation, training on hours-long clinical video becomes prohibitively expensive, and standard LSTMs degrade on very long-range dependencies through gradient issues. RNN-based temporal components nonetheless appear in at least 26 reported surgical-workflow systems, a footprint the review attributes to initial adequacy rather than enduring suitability.

Temporal convolutional networks answered the parallelization bottleneck directly. TeCNO, introduced at MICCAI 2020, brought a multi-stage TCN performing hierarchical prediction refinement to surgical phase recognition, replacing recurrence with dilated causal convolutions so receptive-field size decouples from sequential computation. The result was online inference with smooth predictions across ambiguous phase transitions, matching or exceeding LSTM accuracy on Cholec80 while removing the training-time penalty. Attention-based architectures then pushed further: Trans-SVNet aggregates pre-computed spatial and temporal embeddings rather than attending over raw frames, keeping inference fast enough for interactive use at a reported 91 frames per second on Cholec80 while improving accuracy. Surgformer, presented at MICCAI 2024, extends the line with a Surgical Transformer using hierarchical temporal attention and multi-scale mechanisms to capture dependencies at different temporal granularities, addressing variable phase durations.

Self-supervised pre-training is reshaping the data economics. The Multi-view Masked Contrastive Representation Learning framework, published at NeurIPS 2024, outperformed state-of-the-art self-supervised endoscopic pre-training by 3.5% F1 for classification, 7.5% Dice for segmentation and 2.2% F1 for detection over Endo-FM. Its mechanism, time contrastive learning, synthesizes spatiotemporal features directly from unlabelled video, which matters in a domain where dense annotation is costly. The review frames the trend as a shift away from supervised feature engineering toward pre-trained representations that generalize across surgical tasks with minimal labelled data.

Multi-modal fusion rests on complementarity rather than data volume. Intraoperative environments generate endoscopic video, robotic kinematics, force and tactile measurements, physiological signals and pre-operative imaging, each with its own sampling rate, noise profile and failure mode, so fusion is a temporally aligned coordination problem. Video captures scene semantics that kinematics cannot, while kinematics encode fine motor signals often ambiguous or occluded in video; graph-based models integrating both outperform unimodal baselines specifically on tasks where either modality alone is insufficient. The action-triplet formulation reframes understanding as who is doing what to which structure: the CholecT50 dataset annotates 100 triplet classes spanning 6 instruments, 10 verbs and 15 targets, and in the CholecTriplet2021 challenge mean average precision ranged from 4.2% to 38.1% across 19 algorithms, quantifying how far fine-grained instrument-tissue understanding lags coarse phase recognition, with the binding constraint in verb-target interaction.

Latency requirements, the review argues, are routinely conflated. Three regimes should be distinguished. In teleoperation and robotic control loops, controlled experiments place ideal end-to-end latency at no more than about 200 ms, with roughly 300 ms still usable but degrading precision and delays beyond about 500 ms rendering complex tasks unsafe; stricter dynamic-task studies put the acceptable bound nearer 100 ms, while a deployed telesurgical platform achieved 27 ms against an experimentally determined 125 ms threshold. For video pass-through, a controlled laparoscopy study found significant performance and usability deterioration at 105 ms of added latency, about 160 ms total system latency, with experienced surgeons more affected than trainees. For AI inference on the video stream itself, the model need only keep pace with the roughly 25 fps acquisition rate; a clinically deployed real-time detection system operated at at least 25 frames per second with a latency of 76.80 plus or minus 5.60 ms that endoscopists found imperceptible. The frequently quoted sub-100 ms target, the authors contend, mixes these regimes.

Edge deployment is governed by an accuracy-latency-size trilemma in which no single compression technique suffices, and the reported gains are best read as composable stages. Quantization, the most mature route, routinely shrinks models roughly fourfold moving from 32-bit to 8-bit weights with sub-1% accuracy loss; 4-bit methods such as QLoRA finetune a 65-billion-parameter model on a single 48 GB GPU versus more than 780 GB for full 16-bit finetuning. Pruning yields hardware-friendly dense tensors: SliceGPT slices roughly 25% of parameters from LLaMA-2 70B while retaining about 99% of dense zero-shot performance, with a measured 16-17% inference speed-up on a 24 GB consumer GPU. Knowledge distillation transfers teacher behavior into smaller students; in the original formulation a distilled single model recovered most of a 10-model ensemble’s accuracy on a speech task, 58.8% versus 58.9% frame accuracy. The review notes these figures come from general LLM and vision benchmarks and indicate order of magnitude, not surgical-domain measurements.

Hardware platforms and honest benchmarking occupy the review’s most cautionary pages. NVIDIA reports Clara Holoscan pipelines reaching 10 ms on 4K 240 Hz streams against 220 ms for traditional GStreamer/OpenGL endoscopy pipelines, and a real-world surgical application with median frame rates near 13.5 Hz and end-to-end latency below 75 ms, a 50% reduction over conventional approaches. Independent evaluation is emerging: an external real-time-systems analysis derived worst-case response-time bounds within 0.3% of measured values, and a peer-reviewed though NVIDIA-authored study reported deterministic latencies reduced from 133.8 ms to 90.6 ms, below 50 ms for up to three concurrent endoscopy applications. The counterweight is a clinician-led benchmark deploying surgical phase recognition on an embedded Jetson Xavier NX, which reported approximately 250 ms per prediction at 59.8-78.2% accuracy, underscoring that real-world edge performance often falls well short of vendor figures. Efficient detection architectures tell a similar story: the migration from YOLOv7x to YOLO-MobileNet moves down the parameter curve from 71.3 million to 12.4 million parameters while improving both speed and reported mAP, and InstrumentNet uses YOLOv7 with multiscale and adaptive feature-weighted fusion for real-time intracranial instrument segmentation.

Generalization may be the field’s hardest constraint. Phase- and step-recognition models trained at one hospital for laparoscopic gastric-bypass surgery lost a large fraction of their F1 score when transferred to another without adaptation, recovering only when trained on multi-centre data. The review frames the interaction of accuracy, latency and generalization as a trilemma: long-range attention scales quadratically with sequence length, each added modality enlarges the compression burden, and a temporally deep multimodal model optimized on one institution’s data and compressed for its hardware is triply specialized. Benchmark reliance compounds the risk, with Cholec80 used in 38 reported studies, though newer datasets such as AutoLaparo, 21 full-procedure hysterectomy videos totalling 1,388 minutes with three integrated tasks, and HeiChole, with phase, instrument-presence and skill annotations, were designed to widen external validity. Evaluation protocols themselves vary, including popular relaxed-boundary metrics whose definitions differ across implementations.

Clinical translation remains the exception. AI-Endo, an intelligent workflow recognition system for endoscopic submucosal dissection trained on 201,026 labelled frames and integrated directly with Olympus endoscopic systems for animal-study validation, stands among the clearest exemplars, and the review argues the engineering challenge shifts from optimizing predictions to guaranteeing behavior when predictions are unavailable, making robust fallback protocols a precondition for clinical use. Regulatory data reinforce the gap: the FDA’s AI/ML-enabled device list held roughly 950 entries by mid-2024 and surpassed 1,000 across 2024-2025, with an independent taxonomy cataloguing 1,016 authorizations covering 736 unique devices as of December 2024, 84.4% image-based and 14.5% signal-based, radiology leading review panels at 88.2% while surgical specialties do not feature among the leading panels. Clinical performance studies were reported for only about half of approved devices, 505 of 903, with only 8.1% of reported studies prospective and 2.4% randomized. In January 2025 the FDA published draft guidance on lifecycle management for AI-enabled device software functions. The review’s throughline is that real-time intraoperative AI is a co-design problem, with architecture, modality set, compression strategy and validation protocol chosen jointly against a single deployment specification, and that foundation models and vision-language systems such as EndoChat, built on the Surg-396K instruction dataset, point toward documentation-capable assistants whose near-term value lies in label-efficient transfer rather than headline accuracy on saturated single-centre benchmarks.

Subject of Research: Real-time artificial intelligence architectures for intraoperative surgical workflow recognition

Article Title: Real-time AI architectures for intraoperative surgical workflow: from temporal modeling and multi-modal fusion to efficient edge deployment

Article References: Agrawal, N., & Kharat, M. (2026). Real-time AI architectures for intraoperative surgical workflow: from temporal modeling and multi-modal fusion to efficient edge deployment. BMC Artificial Intelligence, 2(1), Article 22. https://doi.org/10.1186/s44398-026-00038-5

Image Credits: AI Generated

DOI: 10.1186/s44398-026-00038-5

Keywords: surgical AI, phase recognition, temporal modeling, transformers, multi-modal fusion, edge deployment, model compression, Cholec80, FDA regulation, vision-language models, latency, generalization