Mixed reality makes hidden steps of brain surgery planning measurable

Neurosurgeons weigh a dense web of choices before a patient ever reaches the operating room: where to enter the skull, which anatomical structures to protect, and how to balance tumor access against the risk of neurological injury. A new exploratory study from researchers in Austria now shows that a carefully instrumented hybrid of desktop software and extended reality (XR) can capture parts of this hidden process as hard, quantitative data.

The study, published in Frontiers in Digital Health, was led by Anna Drechslerová of the Carinthia University of Applied Sciences together with colleagues at the Medical University of Vienna, including neurosurgeon Georg Widhalm. The team built a synchronized planning framework that links a conventional image-based planning environment, implemented as a custom module in 3D Slicer, with an immersive XR workspace rendered on a Microsoft HoloLens 2 headset. A Python-based TCP server kept the two environments in constant communication, exchanging structured JSON messages so that anatomical structure visibility, model transparency, slice-plane positions, and camera rotations stayed aligned across both views.

Participants could move freely between the desktop workspace, which offered standard two- and three-dimensional image review and planning-parameter adjustment, and the immersive XR view, where patient-specific anatomical models could be inspected from within, manipulated directly, and probed with specialized tools. Three interaction tools supported spatial assessment: a probe-based tool that carved an inspection corridor from a user-defined marker toward the center of the head model, a cross-sectional tool whose cutting plane was synchronized with the corresponding slice in 3D Slicer, and an entry-region planning tool that let users place a preferred surface entry marker and inspect the resulting entry-to-tumor direction through stepwise model cutouts.

Twenty volunteers took part: eight neurosurgeons, six neurosurgical residents, and six medical students, recruited deliberately across levels of clinical experience. After standardized training on a separate non-study case, each participant planned two patient-specific brain tumor cases, one glioblastoma (GBM) and one low-grade glioma (LGG), in counterbalanced order. The cases were retrospective surgical cases already approved for surgery, selected by an experienced neurosurgeon to present anatomically distinct scenarios involving lesions near vascular and eloquent structures. Participants knew the pathology of each case but never saw the original surgical plans, so their choices were made independently of the real operative decisions.

All 20 participants completed the training session and both evaluation cases without any session-ending technical interruption, and no one was excluded because of technical difficulties. Synchronization between the desktop and XR workspaces remained functional throughout, which the authors treat as the primary feasibility outcome of the study. Beyond that, the framework logged task timing, optional exploration-tool use, and entry-marker placement, while participants entered anatomical selections, priority rankings, confidence ratings, and perceived-risk assessments through the evaluation interface. After both cases, they completed the System Usability Scale, an adapted NASA Task Load Index, and study-specific questionnaires.

The most striking spatial result concerned trajectory agreement. Although participants placed surface entry markers at visibly different locations, the directions from those entry points to the tumor center converged strongly. Directional convergence, calculated as the norm of the mean normalized direction vector across participants, reached 0.881 for the GBM case (95% bootstrap confidence interval 0.686 to 0.987) and 0.965 for the LGG case (95% CI 0.935 to 0.990), with directional dispersions of 0.119 and 0.035 respectively. Mean angular deviation from the case-specific mean direction was 17.6 degrees for GBM versus 11.8 degrees for LGG, though the GBM case contained one large outlier of 157.6 degrees. Median entry-to-tumor distances were 97.6 millimeters for the GBM case and 146.4 millimeters for the LGG case.

The authors emphasize that this divergence between entry-point location and entry direction is itself informative. Because different surface positions can yield nearly identical trajectories toward the tumor, evaluating only the marked entry point may underestimate how similar participants’ planning strategies actually are. At the same time, they caution that directional convergence does not establish that the convergent direction was clinically preferable, since no reference trajectory or independent criterion of optimality was available in the study. The metrics describe geometric similarity of access directions, not surgical correctness.

Anatomical prioritization formed a second behavioral dimension. Across both cases, participants identified a mean of 3.45 structures per session, most frequently blood vessels (77.5 percent), motor cortex (60.0 percent), Broca’s area (55.0 percent), and primary somatosensory cortex (45.0 percent). Broca’s area was selected by every participant in the LGG case but only occasionally in the GBM case, while the GBM case drew more attention to vascular and sensorimotor anatomy. Overall agreement on which structures mattered was fair, with a Fleiss’ kappa of 0.397, ranging from 0.311 in the GBM case to 0.408 in the LGG case. By role, kappa was 0.470 among neurosurgeons, 0.425 among residents, and 0.345 among medical students, with matching patterns in Jaccard similarity and entropy measures. The authors stress that these experience-related patterns are descriptive group observations and cannot be attributed causally to expertise, given the small cohort and differences in prior XR and 3D planning experience.

Workflow data added a third layer. Median time for anatomical structure identification fell from 336.2 seconds in the first case to 219.5 seconds in the second, a statistically significant within-participant reduction with a median drop of 141.2 seconds, and entry-region planning time fell from 90.0 to 76.2 seconds. The authors interpret this as a session-order effect consistent with growing familiarity with the experimental environment rather than improved planning performance. Optional exploration tools were used selectively: six of twenty participants used the slicing mode, with a median interaction time of 279.8 seconds among users, and five used the probe mode, with a median of 79.6 seconds. Subjectively, participants rated spatial relationship interpretation at 5.58 out of 7 and access-direction confidence at 5.28 out of 7, while the median System Usability Scale score of 60.0 signaled clear room for interface refinement. The adapted NASA-TLX indicated low-to-moderate workload at a mean of 3.46 out of 10.

The authors are explicit about the limits of what this demonstrates. With only two patient cases, no desktop-only control condition, and no validation against expert reference plans, clinical criteria, intraoperative observations, or patient outcomes, the captured measures must be regarded as descriptive behavioral descriptors rather than validated indicators of planning quality, expertise, or clinical correctness. The study also did not incorporate functional MRI, diffusion tractography, or magnetic resonance spectroscopy in the standardized planning scenes. What the work establishes instead is a methodological foundation: an XR-supported planning environment can be instrumented to record how people actually plan, converting an otherwise experiential workflow into structured data. Future studies with larger and more diverse clinical cohorts, conventional-workflow comparisons, and validation against expert or clinical criteria will determine whether these observable behaviors carry clinically meaningful information, and whether immersive planning can be integrated into routine multidisciplinary practice without unacceptable workflow burden.

Subject of Research: Instrumented hybrid desktop–extended reality framework for quantifying observable preoperative neurosurgical planning behavior

Article Title: Integrating extended reality into preoperative neurosurgical planning: exploratory evaluation of planning-related behavior in an instrumented hybrid framework

Article References: Drechslerová, A., Widhalm, G., Hangel, G., Rajabi, N., Huber, J., Grabner, G., Bogner, W., Hartl, A. D., & Kiesel, B. (2026). Integrating extended reality into preoperative neurosurgical planning: exploratory evaluation of planning-related behavior in an instrumented hybrid framework. Frontiers in Digital Health, 8, 1978547. https://doi.org/10.3389/fdgth.2026.1978547

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1978547

Keywords: extended reality, mixed reality, neurosurgical planning, preoperative planning, 3D Slicer, Microsoft HoloLens 2, glioblastoma, low-grade glioma, directional convergence, Fleiss kappa, System Usability Scale, surgical workflow