AI Learns to Fly a Twin-Rotor Helicopter in Simulation Using Twin Delayed DDPG

A twin-rotor helicopter teetering on two axes of motion has long been a favorite torture test for control engineers. Its pitch and yaw are tangled together by aerodynamic cross-coupling, so a correction on one axis shoves the other off course, and the whole machine is underactuated, meaning there is less independent control authority than there are degrees of freedom to manage. A new study published in Discover Robotics reports that a deep reinforcement learning controller built on the twin delayed deep deterministic policy gradient, or TD3, algorithm can track demanding reference trajectories on such a 2-degree-of-freedom helicopter in simulation, outperforming classical PID and fuzzy logic benchmarks on overshoot, settling time, steady-state accuracy, and smoothness of the control signal.

The work, led by Zied Ben Hazem with colleagues including Muhammed Özdemir, Firas Saidi, and Nivine Guler, is entirely computational. The team built a high-fidelity MATLAB/Simulink model of the laboratory twin-rotor platform, deriving its nonlinear equations of motion from Newton–Euler mechanics, including gravity moments, centrifugal coupling terms, friction, and first-order actuator dynamics that convert pulse-width-modulated duty cycles into propeller thrust. Crucially, although this physical model was needed to construct the training simulator, the learned controller itself is strictly model-free: the TD3 agent maps observed states directly to continuous motor commands without any explicit knowledge of the equations, feedforward paths, or real-time parameter estimation.

TD3 is a refinement of the earlier deep deterministic policy gradient method, which has a well-known weakness in strongly coupled systems. When independent control loops fail to anticipate cross-axis torque disturbances, the target value estimates swing wildly and the critic networks systematically overestimate action values, which can destabilize or derail training entirely. TD3 counters this with three mechanisms: twin critic networks whose minimum prediction forms the temporal-difference target, suppressing overestimation bias; delayed policy updates that let the critics converge before the actor moves; and target policy smoothing, which adds clipped noise to target actions to prevent overfitting to sharp value spikes.

The reward function is where the authors claim a distinctive design contribution. Rather than penalizing tracking error alone, the reward is structured with principles borrowed from artificial potential field theory, balancing attraction toward the desired trajectory against penalization of deviations and excessive control effort. Weighting coefficients for pitch error, yaw error, and control energy were selected by a grid search, with tracking weights deliberately set higher than the energy term so that accuracy takes priority. The result, the authors report, is a policy that converges efficiently while producing smooth, energy-conscious actuation rather than aggressive, jittery commands.

The benchmarking was thorough. The TD3 agent was pitted against a classical PID controller, a fuzzy PD + I controller, and a fuzzy PD + FF controller with a feedforward term, all previously developed and experimentally validated on twin-rotor systems. Three reference trajectories were used: a multi-amplitude step sequence to probe transient response and cross-coupling, a smooth sinusoid for continuous tracking, and a Lissajous curve chosen specifically to stress the nonlinear pitch–yaw coupling. Performance was scored with overshoot, peak time, 5% settling time, steady-state error, root-mean-square error, integral error indices, and the total variation of the control signal as a smoothness measure.

The numbers are striking, with the caveat that they come from simulation. On the multi-step trajectory, TD3 cut maximum overshoot to 4.6% in pitch and 3.8% in yaw, against 22.4% and 18.7% for PID, and reduced settling time by up to 66%. Steady-state error fell to 0.18 degrees in pitch and 0.15 degrees in yaw, improvements the authors put at more than 80% over PID. Cross-coupling effects were reduced by over 60%, and the control signals showed the lowest total variation of any controller tested, implying less actuator wear. Similar orderings held on the sinusoidal and Lissajous trajectories, where TD3 kept induced pitch–yaw deviations under roughly 2 to 2.5 degrees.

Robustness testing added further pressure. The team injected internal disturbances as ±10% variations in rotor thrust coefficients and ±15% variations in effective pitch and yaw inertia, applied persistently, alongside external torque pulses of 10–20% of nominal maximum control torque lasting 0.2 to 0.5 seconds, mimicking wind gusts and payload shifts. Identical disturbance profiles were applied to every controller. Under these conditions, TD3 consistently recorded the lowest error metrics, the fastest recovery, and the smoothest control action, reducing control variation by up to 65% compared with the conventional methods, while PID showed pronounced oscillations and slower recovery.

An ablation study backs up the claim that the gains come from TD3’s specific machinery rather than deep learning in general. By systematically isolating twin critics, delayed policy updates, and target policy smoothing against a standard DDPG baseline, the authors attribute the performance improvements directly to these algorithmic components. They are also candid about limits: TD3 improves learning stability through its mechanisms but does not provide formal closed-loop stability guarantees in the classical control sense, so stability was assessed empirically across operating conditions. The authors likewise acknowledge that reinforcement learning remains sample-inefficient, that no explicit latency compensation is built into the current design, and that the learned policy may degrade when pushed into operating regimes far outside its training distribution.

The comparison with recent state-of-the-art paradigms frames the practical appeal. Robust deterministic controllers augmented with neural approximation layers offer stability guarantees but often demand model-dependent computations at every runtime step, while online adaptive reinforcement learning approaches impose computational overhead that can be prohibitive for embedded hardware. The proposed framework instead relocates the expensive value iteration and policy optimization to offline simulation, so the deployed controller reduces to a lightweight feedforward inference network requiring no real-time gradient descent or recursive parameter updates.

For now, the caveat matters: this is a validated simulation study, not a flight test, and the authors themselves identify hardware validation on an experimental twin-rotor rig as the immediate and essential next step, with sensor noise, communication delays, unmodeled actuator dynamics, and safety during real-time execution all awaiting confrontation. Still, the results establish a credible case that a model-free TD3 policy can learn precise, decoupled, disturbance-tolerant control of a notoriously awkward underactuated aerial platform, and the authors sketch extensions to quadrotors via cascade architectures and hybrid fuzzy–reinforcement learning frameworks to accelerate exploration. If the simulation-to-hardware gap closes as hoped, the approach could matter for autonomous inspection, surveillance, and adaptive flight in uncertain environments.

Subject of Research: Deep reinforcement learning control of a 2-DOF helicopter for trajectory tracking

Article Title: Deep reinforcement learning for accurate trajectory tracking in 2-DOF helicopter dynamics using twin delayed DDPG

Article References: Ben Hazem, Z., Özdemir, M., Saidi, F., & Guler, N. (2026). Deep reinforcement learning for accurate trajectory tracking in 2-DOF helicopter dynamics using twin delayed DDPG. Discover Robotics, 2(1), Article 23. https://doi.org/10.1007/s44430-026-00035-3

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00035-3

Keywords: deep reinforcement learning, TD3, 2-DOF helicopter, trajectory tracking, underactuated systems, control engineering, twin-rotor, cross-coupling, robustness, simulation, PID controller, fuzzy control