Anyone who has battled an arcade claw machine knows that success depends on more than simply moving a joystick. Players must mentally translate each push or pull into the motion of a mechanical arm, estimate how the claw will move through three-dimensional space, and coordinate several controls at once. Excavator operators face a similar challenge, but on a far larger and more consequential scale. Instead of grabbing a toy, they manipulate tons of earth, rock, and debris using a boom, arm, bucket, and rotating cab. Now, engineers at MIT have developed a robotic control interface designed to make operating an excavator feel less like learning a complex video game and more like moving the machine’s own arm with your body.
The new system, called the World-Space Interface, replaces conventional excavator joysticks with a miniature mechanical arm and bucket. An operator grips the device and moves it in a way that directly resembles the intended motion of a full-size excavator. The operator’s movements are tracked and translated into commands for a virtual excavator displayed on an immersive six-screen system. When the trainee raises, lowers, or rotates the miniature arm, the simulated machine mirrors those actions inside a digital construction environment. The system is designed to reduce the mental translation traditionally required to connect joystick inputs with the movement of the excavator’s boom, stick, bucket, and cab.
“This is a more intuitive way to command the machine,” says Hermano Krebs, principal research scientist in MIT’s Department of Mechanical Engineering. According to Krebs, traditional controls force operators to build an internal “mental map” of how joystick movements correspond to the machine’s physical actions. That map becomes especially complicated because excavator controls can vary depending on the operator’s position, the machine’s configuration, and the control pattern selected. With the World-Space Interface, the physical controller imitates the excavator itself, allowing users to gesture toward the desired movement instead of remembering which joystick direction produces it.
An excavator is essentially a coordinated system of hydraulic actuators and articulated joints. The boom connects the main body of the machine to the arm, or stick, while the stick supports the bucket. Hydraulic cylinders extend and retract to change the angle of each component, and the cab and upper structure can rotate relative to the tracked undercarriage. A skilled operator must combine these motions smoothly to scoop material, swing the load, position it accurately, and release it without damaging surrounding structures or wasting energy. The World-Space Interface maps the miniature mechanism’s movements to these corresponding joints in a virtual model, enabling trainees to practice the coordination required for real excavation without immediately exposing them to the risks and costs of operating heavy equipment.
The project grew from a collaboration between Krebs’ human-robot interaction group and Sumitomo Heavy Industries, a Japanese manufacturer of industrial machinery. The collaboration began in 2018, when researchers identified a growing challenge in Japan: many heavy-equipment operators were approaching retirement, while training replacements could require months or years of experience. Conventional training typically takes place on a controlled driving course using real excavators. Although this approach provides valuable practical experience, it is expensive, physically demanding, and potentially hazardous for beginners. A simulator can provide a safer environment, but most existing systems continue to rely on the same joystick arrangements that novices must eventually learn on actual machines.
To test whether a more physically intuitive interface could accelerate learning, the MIT team created 15 virtual excavation environments. The simulated settings included construction sites, highways, forest roads, riverbanks, mining areas, and urban and rural landscapes. Participants performed tasks modeled on real industrial work, including scooping and dumping sand or gravel, digging and grading trenches, clearing roadside debris, removing branches from the edges of waterways, and breaking apart rocks. These tasks required users to control multiple excavator functions at the same time while maintaining awareness of the surrounding environment and the position of the bucket.
Volunteers trained for one hour per day over seven days using both the World-Space Interface and a conventional joystick-based simulator. The experiment included experienced operators and people with little or no excavator experience. The researchers compared performance before and after the training period, examining how effectively participants completed simulated tasks as their difficulty increased. On the joystick simulator, novices initially performed worse than experts, although their performance improved with practice. The difference between the groups reflected the learning burden imposed by the control system: beginners had to acquire the machine’s control logic before they could focus fully on the excavation task itself.
The results were notably different with the World-Space Interface. Novice participants performed at a level comparable to experienced operators from the beginning of the experiment. The finding suggests that the physical resemblance between the controller and the excavator’s working arm may allow users to transfer ordinary body-based movement skills directly into machine operation. Rather than thinking, for example, about which control must be moved to lower the boom while another control rotates the cab, a trainee can move the miniature arm in the direction and configuration associated with the intended action. “This is the first interface that does not require me to command the excavator with joysticks,” says study co-author Joao Buzzatto, an MIT postdoctoral researcher.
The researchers describe the system as a “world-space” controller because it allows the operator to manipulate objects outside the cab through movements that correspond directly to actions in the surrounding environment. In a conventional excavator, the operator sees the work area through the cab windows but controls the machine through hand movements that may have no obvious physical relationship to the bucket’s motion. The World-Space Interface instead links the operator’s hand position to the excavator’s visible action in the virtual world. The team is now developing a haptic version that could add physical feedback to the miniature arm. If a simulated bucket encounters resistance or lifts a heavy load, the controller could generate force against the operator’s hand, creating the sensation of weight, contact, or obstruction. Such feedback could help trainees understand the machine’s interaction with soil and materials, while also making remote operation more precise.
MIT researchers envision several possible applications beyond classroom training. The miniature arm could be installed inside an excavator cab as an alternative to joysticks, allowing operators to command the machine through a more natural movement interface. It could also be connected to an excavator operating in a dangerous or inaccessible location. In that scenario, an operator might sit inside a trailer or control center and use the mechanical arm to tele-operate a machine from a safe distance. The approach could be useful in unstable construction zones, disaster areas, contaminated sites, mines, or locations where falling debris and difficult terrain place workers at risk. Companies including Caterpillar, Hyundai, and Komatsu are already developing virtual and remote-operation systems, but the MIT team argues that many remain limited by joystick-based controls. By making the controller resemble the machine’s own anatomy, the World-Space Interface could help new operators begin productive work sooner, potentially changing how heavy equipment is learned and controlled.
Subject of Research: An intuitive mechanical-arm controller and virtual-reality training platform for excavator operators.
Article Title: A Comprehensive Training Platform for Excavator Operators: Training with Joint-Space and World-Space Interfaces in Virtual Environments
News Publication Date: 17-Aug-2026
Web References: https://doi.org/10.1061/JCCEE5.CPENG-7488
References: Journal of Computing in Civil Engineering, “A Comprehensive Training Platform for Excavator Operators: Training with Joint-Space and World-Space Interfaces in Virtual Environments,” DOI: 10.1061/JCCEE5.CPENG-7488
Keywords
Excavators, robotics, human-machine interaction, virtual reality, operator training, teleoperation, haptic feedback, construction technology, mechanical engineering, construction equipment
Tags: advanced operator training technologydigital twin technology for construction machineryergonomic control systems for excavator operatorshuman-machine interaction in construction equipmentimmersive multi-screen construction simulationimmersive virtual excavator training systemjoystick replacement with miniature mechanical armMIT engineering innovations in heavy machinery controlprecision control of large-scale construction equipmentreal-time motion translation for excavator operationRobotic control interface for construction excavatorsWorld-Space Interface for heavy machinery
