Deep Reinforcement Learning Teaches Autonomous Cars to Change Lanes Safely and Smoothly

Autonomous vehicles promise to transform road travel, but the technology’s viability ultimately hinges on two intertwined qualities: driving efficiency and driving safety. Few maneuvers test both qualities as relentlessly as the lane change. Performed constantly in dynamic traffic flows on highways and city streets, the lane change demands that a vehicle predict what surrounding cars will do next, weigh collision risk against progress toward its destination, and execute a trajectory that is smooth enough to keep passengers comfortable. A new study from researchers at Hunan Mechanical and Electrical Polytechnic in Changsha, China, published in Frontiers in Mechanical Engineering, tackles this challenge with an improved deep reinforcement learning framework that jointly optimizes trajectory prediction and lane-changing decision-making, achieving obstacle avoidance success rates of 98.92 percent and target advancement success rates of 98.26 percent in simulation—figures that outperform a suite of established comparison algorithms.

The research team, Yuexiang Hu and Qin Li, identified two persistent weaknesses in existing lane-changing trajectory planning methods: low prediction accuracy and insufficient multi-objective collaborative optimization. Many current approaches either cascade a prediction module with a decision-making network in a modular assembly, where the two components are trained independently and gradient flow is truncated at the interface, or they rely on manually designed state spaces and reward functions that transform abstract safety constraints into rigid rules. In such architectures, the predictor may generate trajectories that are statistically accurate yet strategically suboptimal for lane-changing decisions, because the prediction module optimizes only for coordinate regression accuracy while remaining unaware of decision-critical factors such as collision risk or comfort requirements. The new framework eliminates this modular boundary by constructing a unified training objective where prediction and control are co-trained under shared gradient flows, so prediction errors directly influence policy gradients and the policy network can adapt its state representation based on prediction reliability.

At the heart of the trajectory prediction module lies a carefully engineered combination of three neural network components: a temporal convolutional network (TCN), a spatial multi-head attention mechanism, and a Transformer encoder–decoder. The TCN employs three stacked dilated causal convolution layers with a two-dimensional kernel size of (7,1), extracting multi-scale temporal dependencies from historical vehicle states through exponentially increasing receptive fields. Layer normalization and ReLU activation follow each TCN layer to stabilize training dynamics. The spatial attention mechanism operates on a dynamic spatiototemporal graph where nodes represent vehicle states, temporal edges connect consecutive historical states across a 10-second observation window, and spatial edges encode instantaneous interactions between vehicles within a perception range extending 50 meters ahead, 30 meters behind, and across the full lateral coverage of adjacent lanes. Eight attention heads with a model dimension of 32 extract different interaction patterns in parallel, with attention weighting based on relative motion states rather than fixed positional encoding—enabling the network to adaptively model inter-vehicle topological interactions through the dynamic spatiototemporal graph.

The Transformer encoder–decoder consists of six encoder layers and six decoder layers, processing spatiototemporal features of variable-length historical encoding and multi-step future decoding within a 5-second prediction range. Dropout at 0.11 prevents overfitting, and the network trains using the Adam optimizer with a learning rate of 0.0001, with the output layer employing linear activation to directly regress future trajectory coordinates. To quantify potential collision risk during lane changes, the researchers built a driving risk field model comprising three components with specific parameters: static obstacle risk following Gaussian decay with an influence radius of 40 meters and a decay coefficient of 10 meters; dynamic obstacle risk with directional correction using an approach factor of 0.5 and a recede factor of −0.3; and road boundary risk following a power function with a boundary coefficient of 2.0. Lane line risk uses a Gaussian-like function with a deviation penalty of 0.5 m⁻² and a peak risk of 1.0, with field strength set to 1.0 for solid lines and 0.5 for dashed lines. Virtual mass characterizing the risk field is 1,500 kilograms, reflecting a typical passenger vehicle.

With the prediction module and risk field in place, the researchers turned to the deep deterministic policy gradient (DDPG) algorithm—a deep reinforcement learning method prized for efficiently processing continuous action spaces and enabling end-to-end optimization. DDPG employs four neural networks: an online policy network (Actor) that outputs deterministic actions, an online value network (Critic) that estimates the Q value of state–action pairs, and their corresponding target networks used to calculate stable target Q values and reduce sample correlation. The Actor network receives a 13-dimensional state vector—comprising self-vehicle state, nearest surrounding vehicle state, predicted trajectories, and driving risk field values—and outputs three continuous control actions (throttle, steering, and brake) through a Tanh-activated output layer. The Critic network evaluates the Q value of the state–action pair using a linear output layer, providing gradient guidance for policy optimization. Target networks soft-update at each training step with a factor of 0.004 to stabilize temporal difference error estimation, while Ornstein–Uhlenbeck noise with parameters (0.14, 0.00, 0.24) superimposes on the Actor output for continuous action–space exploration.

One of the study’s most consequential improvements concerns the experience replay pool. Traditional DDPG uses uniformly random sampling, treating all experiences equally—a design that dilutes key safety experiences such as emergency obstacle avoidance and close-range car following beneath a mass of conventional lane-changing data, slowing convergence and reducing decision-making safety. The improved mechanism introduces a priority screening based on the temporal difference error: newly collected experiences deposit into a collection pool, and those with temporal difference errors exceeding a threshold of 0.10 filter into a high-quality pool for prioritized batch sampling. This ensures that high-value safety experiences obtain higher sampling probability, effectively solving the problem of important experiences being submerged. The reward function guiding the agent comprises four weighted components: efficiency reward (weight 0.3) encouraging forward travel distance, comfort reward (weight 0.2) penalizing sudden acceleration and steering angle changes, safety reward (weight 0.4) directly adopting the negative risk field intensity so higher risk yields greater penalty, and driving state reward (weight 0.1) providing positive rewards for normal driving and negative rewards for lane departure or collision. A sensitivity analysis confirmed that the nominal configuration achieves a balanced trade-off among safety, efficiency, and comfort, maintaining a collision rate of 3.27 percent and lane departure rate of 2.36 percent while ensuring reasonable velocity.

The experimental data came from the NGSIM US-101 vehicle trajectory public dataset, with trajectories divided into a complete observation period of 15 seconds—the first 10 seconds serving as historical input features and the last 5 seconds as output labels to predict. Savitzky–Golay filtering removed outliers, and after preprocessing 14,341 pieces of data were obtained, split 70 percent for training and 30 percent for verification. When compared against Structural-LSTM, Social-GAN, and constant turn rate and acceleration (CTRA) models, the proposed prediction module achieved root mean square errors (RMSE) of 1.59 meters for going straight, 1.19 meters for turning left, and 1.27 meters for turning right—substantially lower than the 2.08, 1.51, and 1.98 meters of Structural-LSTM, the 2.47, 1.62, and 1.98 meters of Social-GAN, and the 2.86, 1.98, and 2.86 meters of CTRA under corresponding conditions. Repeated 10 times with different random seeds, these results held as mean ± standard deviation, with paired t-tests between the proposed method and each baseline yielding p < 0.01, indicating statistically significant improvements.

Ablation experiments confirmed the contribution of each module. Removing the TCN module led to the largest increase in prediction RMSE by 0.62 meters, indicating its critical role in temporal feature extraction. Without the attention mechanism, obstacle avoidance success rate dropped by 2.71 percent. The risk field contributed most significantly to safety—its removal caused a 13.25 percent drop in obstacle avoidance success rate—while the improved replay mechanism accelerated convergence, with its absence resulting in a 15.69 percent decrease in obstacle avoidance success rate. The simultaneous removal of the risk field and replay mechanism caused the most severe performance degradation by 20.47 percent and 19.03 percent, demonstrating their joint importance. In simulation experiments on the CAR Learning to Act platform with 200 obstacles and targets randomly arranged, the proposed method achieved obstacle avoidance success rates of 98.92 ± 0.35 percent and target advancement success rates of 98.26 ± 0.42 percent, outperforming LSTM-DDPG (94.37 ± 0.52 and 92.21 ± 0.58), TRPO (90.26 ± 0.68 and 89.87 ± 0.62), and TD3 (89.24 ± 0.71 and 88.64 ± 0.65), with paired t-tests again yielding p < 0.01.

Beyond raw success rates, the method demonstrated superior trajectory quality in overtaking and obstacle avoidance scenarios. In lane-changing overtaking, all algorithms completed the left lane change in approximately 10 seconds, but the proposed method completed the right reversal operation in 45 seconds compared with approximately 53 seconds for LSTM-DDPG, while TRPO and TD3 stayed in the original lane entirely. The proposed method maintained stable vehicle speed throughout and quickly increased speed after overtaking, indicating higher overtaking efficiency. In obstacle avoidance, the maximum lateral displacement of the proposed method was −3.18 meters, and it quickly returned to the positive direction after 40 seconds with stable movement trajectory—compared with −3.76, −3.47, and −3.62 meters for LSTM-DDPG, TRPO, and TD3, which showed slower back-to-alignment speed and poorer trajectory control accuracy. The proposed method maintained vehicle speed at 57.87 km/h during obstacle avoidance with the smallest fluctuation, and its steering angle control curve was smoother with fewer peaks, indicating better comfort and stability.

Perhaps most strikingly, the method achieved an inference time of 19.8 ± 1.5 milliseconds per step, corresponding to a frame rate of 50.5 ± 3.8 Hz—meeting the real-time requirement for autonomous driving applications—while consuming the lowest GPU memory at 95.2 ± 6.4 MB. Comparison methods including SAC, PPO, Graph-based, Social-LSTM, LSTM-DDPG, TRPO, and TD3 exhibited inference times ranging from 35.4 to 52.4 milliseconds and memory consumption from 108.6 to 156.8 MB. Robustness testing under challenging conditions further validated the approach: in dense traffic with 8 to 12 surrounding vehicles exhibiting aggressive behaviors including frequent lane cut-ins, reduced headways of 1.5 seconds, and elevated acceleration up to 6 m/s², the proposed method achieved obstacle avoidance success rates of 95.34 ± 0.48 percent and target advancement success rates of 94.87 ± 0.52 percent. In rainy weather with precipitation intensity of 15 mm/h, road friction coefficient reduced from 0.9 to 0.4, perception range limited from 50 to 35 meters, and Gaussian noise with standard deviation of 0.3 meters added to position and velocity measurements, the method attained obstacle avoidance and target advancement success rates of 93.18 ± 0.55 percent and 92.65 ± 0.61 percent—exceeding all baselines under both scenarios. The researchers acknowledge three principal limitations: experiments were confined to the CAR Learning to Act simulation platform, leaving sim-to-real transfer challenges including domain discrepancy between simulated and real-world sensor inputs, unmodeled actuator latency and steering dynamics, and divergence between idealized traffic behaviors and intricate human driving maneuvers; the model was trained and evaluated on a single dataset with fixed simulation settings, leaving generalization performance on unseen road geometries, diverse traffic rules, and rare edge cases unassessed; and reward weights and risk field parameters were manually calibrated under nominal conditions, warranting future exploration via online meta-learning or domain randomization. Future work will focus on constructing a real-vehicle test platform, incorporating vehicle–road cooperative perception data to mitigate the sim-to-real gap, and introducing domain randomization and adversarial training during simulation to improve policy robustness prior to physical deployment.

Subject of Research: Multi-objective optimization of lane-changing trajectories for autonomous vehicles based on deep reinforcement learning

Article Title: Multi-objective optimization of lane-changing trajectories for autonomous vehicles based on deep reinforcement learning

Article References: Hu, Y., & Li, Q. (2026). Multi-objective optimization of lane-changing trajectories for autonomous vehicles based on deep reinforcement learning. Frontiers in Mechanical Engineering, 12, 1900959. https://doi.org/10.3389/fmech.2026.1900959

Image Credits: AI Generated

DOI: 10.3389/fmech.2026.1900959

Keywords: autonomous vehicles, deep reinforcement learning, DDPG, lane-changing trajectory planning, Transformer, driving risk field, trajectory prediction, Frontiers in Mechanical Engineering, Multi-objective, optimization, lane-changing, trajectories