Mixed reality office reads bodies and moods to reshape virtual space

A mixed reality office prototype that watches how people sit, move, focus and feel, and then reshapes the virtual space around them in real time, has been described by researchers in the journal Architectural Intelligence. The system, built around a smart office design called SENSE BOX, replaces the usual trigger-based model of mixed reality interaction, in which a specific gesture or action calls up a predefined response, with a framework that continuously senses behavior across four scales and translates it into parallel spatial changes.

The work, published on 4 August 2026 by Menghang Liu, Ruyi Yang, Luqi Feng, Yanzi Jin, Wenjie Wang and Chao Yan, addresses a gap the authors identify in current mixed reality environments. Most systems today follow what the researchers call a trigger-based input-feedback loop: an isolated action involving the head, hands or fingers is detected, interpreted, and answered with pre-programmed changes in the virtual scene. The team argues that this approach overlooks the interrelated nature of human behavior, which derives its meaning from patterns rather than isolated movements, and may leave interactions feeling disjointed, weakening the sense of presence.

The framework rests on three components: perception, computation and effect generation. A ZED depth camera positioned in front of the installation captures the scene and generates depth maps, which the ZED SDK processes to extract the three-dimensional coordinates of 34 human skeletal keypoints, including the head, shoulders, elbows and knees. Every 0.5 seconds, these skeletal coordinates are transmitted to Rhino Grasshopper over the UDP protocol. The skeleton tracking relies on a deep-learning pose inference algorithm that estimates keypoint positions from depth information and stays reliable under minor occlusions, such as when a person wears a head-mounted display. Images from the same camera are also sent to the Google Vision API, which detects facial features.

From this data stream, the system computes four behavioral states. At the social scale, head and eye coordinates of everyone in the scene are used to calculate head orientations; if the angle between each person’s head-facing vector and the line connecting their head coordinates falls below a threshold of 30 degrees, two people are judged to be collaborating. That threshold came from pilot testing in which participants rotated away from each other until the interaction no longer appeared face-to-face, with the observed boundary at roughly 30 degrees. The collaboration state is then combined with relative distance to sort pairs into four scenarios, from close together and collaborating to far apart and not collaborating, each capable of triggering a different spatial effect.

At the global-postural scale, the system evaluates seated posture through joint angles. Knee flexion angle and trunk-thigh angle are compared against a 90-degree reference, grounded in occupational ergonomics, where joint angles near 90 degrees at hips and knees are associated with minimal musculoskeletal load and a stable postural configuration. Deviations from that reference are normalized into a lower-body physical state score. Shoulder abduction angle and elbow flexion angle are compared against 180 degrees, reflecting the fully extended, relaxed upper limb position consistent with the scoring logic of the Rapid Upper Limb Assessment, and averaged into an upper limb score. Both scores run from 0 to 1, with higher values indicating more flexible or effortful postures.

The local-gestural scale tracks attention through head stability. Once per second, the system computes the angular difference between the current head-orientation vector and the one from the previous second. Based on a small pilot test showing that natural head micro-movements typically fall within about 2 to 3 degrees, the team set a threshold of 3 degrees; differences below it are recorded as 1, larger ones as 0. The average of the five most recent values over a 5-second window yields an attentional state score, where higher values indicate focus and lower values distraction. At the facial scale, the Google Vision API analyzes frames at 0.5-second intervals and returns likelihood values for happiness, sadness, anger and surprise, which are mapped numerically, normalized to a 0 to 1 range, and averaged across detected faces to represent the emotional state.

These states then drive the virtual environment. The space is first modeled parametrically in Rhino Grasshopper, where points form planes and planes are stitched into spatial enclosures and converted to meshes. Using the Rhino.Inside.Unity workflow, the geometry and behavioral parameters pass directly into Unity in real time and are rendered through the Meta XR SDK on a Meta Quest 3 headset. Rather than merging the states into a single composite index, each one independently controls a dedicated set of spatial parameters. The lower-body physical state controls spatial openness across four intervals, from most enclosed to most open; the upper limb state controls spatial regularity, from least structured to most structured; the attentional state controls ceiling height, excluding the area directly above the user’s head, so that greater focus shrinks the perceived space and draws it closer to the body; and the emotional state controls hue and saturation, with happiness, the most sensitive and stable indicator among the four emotions, mapped to red whose saturation rises with the detected value.

The social state produces adaptive partitioning. When two people sit on opposite sides of the space, the system decides whether to generate partitions for privacy when someone prefers undisturbed work, or to adjust them to encourage face-to-face communication when collaboration is detected. The physical prototype consists of bent steel pipes with seats on both sides, a central tabletop and a three-sided enclosing structure, accommodating one to four people, onto which the virtual environment is overlaid. The entire feedback loop runs autonomously and requires no deliberate user input, which the authors say gives users a sense of contextual relevance and adaptability.

To evaluate the experience, the team ran a within-participant study with 20 students from Tongji University, 8 males and 12 females aged 23 to 29, who experienced both the behavior-driven adaptive mode and a manual-trigger mode in which they adjusted the same spatial attributes through five sliders on a desktop interface. Participants rated presence, usability, workload, emotional experience, sense of control and naturalness on 7-point Likert scales. The adaptive mode scored higher on usability (5.87 versus 5.17, a statistically significant difference) and on emotional experience (5.70 versus 5.10, also significant), and trended higher on immersion (5.65 versus 4.85, not significant) and naturalness (5.60 versus 4.83, not significant). The manual mode produced a significantly stronger sense of control (5.90 versus 4.65). Workload and overall satisfaction did not differ significantly, and willingness to use was nearly split: 50 percent found the adaptive mode more suitable for real offices, 55 percent favored it for long-term use, and 70 percent preferred it overall.

The authors are careful about the limits of these findings. The sample was small and homogeneous, participants simulated office states rather than performing authentic work, and the two conditions differed in interface modality as well as adaptiveness, so some differences may reflect embodied sensing versus explicit slider control rather than the adaptive system alone. Open-ended feedback pointed to occasional misclassification of subtle movements, noticeable recognition delay, and a desire for a hybrid mode combining automatic adaptation with optional manual control. The team also notes hardware constraints, including joint dislocations when occlusions block the depth camera and wireless bandwidth limits that cause latency, alongside privacy considerations: raw sensor data is discarded after each processing cycle, but future deployments should explain how behavioral data is interpreted and provide explicit opt-out mechanisms. The authors position the framework as a modular architecture whose sensing layer, multi-scale computation pipeline and feedback mechanism could transfer to other domains, provided behavioral state definitions, thresholds and effect mappings are recalibrated for each new context, and they see integration with IoT-enabled smart offices as a potential next step for more natural, context-aware environmental responses.

Subject of Research: A multilevel computing framework that recognizes human behavior across social, postural, gestural and facial scales to drive adaptive mixed reality spatial interaction

Article Title: A multilevel computing framework for behavior recognition and interaction in mixed reality workspaces

Article References: Liu, M., Yang, R., Feng, L., Jin, Y., Wang, W., & Yan, C. (2026). A multilevel computing framework for behavior recognition and interaction in mixed reality workspaces. Architectural Intelligence, 5(1), Article 24. https://doi.org/10.1007/s44223-026-00127-5

Image Credits: AI Generated

DOI: 10.1007/s44223-026-00127-5

Keywords: mixed reality, behavior recognition, adaptive environments, smart office, human-computer interaction, affective computing, spatial interaction, virtual environments, ergonomics, user study, immersive technology, SENSE BOX