Reinforcement Learning Takes the Wheel of Particle Swarm Traffic Signal Tuning

Traffic engineers have long wrestled with a deceptively simple question: how long should each green light last? A new study published in Frontiers in Artificial Intelligence proposes an answer that pairs a classic swarm-based optimizer with a reinforcement learning agent acting as its strategist. The framework, called PPO-NAMSPSO, uses Proximal Policy Optimization (PPO) to decide, at every iteration of the search, which of six optimization operators should refine a set of candidate traffic signal timing plans. In simulations, the approach reduced average vehicle waiting times by 8.10% on a synthetic grid network and by 32.35% on a realistic urban network derived from OpenStreetMap, both relative to conventional Particle Swarm Optimization (PSO).

The problem the researchers targeted is well known in the traffic control community. Traffic signal timing is a non-linear, stochastic and time-varying optimization challenge, and standard metaheuristics such as PSO, Genetic Algorithms (GA) and Ant Colony Optimization (ACO) typically run one fixed search strategy from start to finish. That rigidity often leads to premature convergence on a poor timing plan, particularly under the shifting demand patterns of a busy day. Earlier approaches, from Webster’s fixed-time calculations through actuated detectors and network-wide adaptive systems like SCOOT and SCATS, each improved on their predecessors, but the authors argue that none dynamically adapts the internal search behavior of the optimizer itself as conditions change.

PPO-NAMSPSO takes an unusual route by placing reinforcement learning at the meta-optimization level rather than using it to control signals directly. At each iteration, a PPO agent observes an eight-dimensional state vector describing the search: optimization progress, normalized swarm diversity, best waiting time, average queue length, average novelty score, stagnation level, recent improvement in the global best, and the operator selection history. Based on that state, the agent picks one of six actions: a standard PSO update only, novelty-guided exploration, an elite-guided update, mutation-based search, a restart strategy, or local search refinement. The standard PSO velocity and position updates provide the base population movement, and the selected action determines which additional search mechanism, if any, is layered on top.

The novelty component is central to the design. Instead of ranking particles solely by waiting time, the framework computes a behavioral descriptor for each candidate plan consisting of its average waiting time and queue length, measures the distance of that descriptor to those stored in a behavior archive capped at 100 entries, and folds a normalized novelty score into a composite fitness function alongside queue length. The final optimization objective, however, remains the minimization of average vehicle waiting time. An elite pool of the top-ranked personal-best solutions guides exploitation, and a stagnation recovery mechanism reinitializes the worst half of the swarm, eight of seventeen particles in the tuned configuration, after four consecutive iterations without improvement in the global best.

Before reinforcement learning begins, the framework uses Optuna with a Tree-structured Parzen Estimator sampler to fix baseline hyperparameters, including a swarm population of 17, an inertia weight of 0.7944, an elite pool size of 3, and weighting coefficients for the queue and novelty terms. The PPO agent itself was trained for 15,000 interaction timesteps using a reward that rewards reductions in global-best waiting time while encouraging swarm diversity, behavioral novelty and avoidance of stagnation, with the reward clipped to the interval from -5 to 5 for training stability. Once trained, the policy was frozen and used purely for inference during evaluation.

Testing took place in the SUMO microscopic traffic simulator, connected to the optimizer through the TraCI interface, on two networks: a synthetic 3 by 3 grid with nine intersections, and an OpenStreetMap-derived urban network with 25 signalized intersections featuring irregular geometries and heterogeneous lane configurations. Two demand scenarios were generated, 1,350 vehicles under medium demand and 2,472 under high demand, with car, bike, bus and truck traffic following a dual-peak distribution over one-hour simulations. Every competing algorithm, PSO, GA, ACO and fixed-time control, received the same Optuna tuning budget of 20 trials, and each optimizer was trained with five random seeds and then evaluated over 100 independent simulation runs.

The results favored the hybrid approach in every reported comparison. On the synthetic grid, PPO-NAMSPSO achieved an average waiting time of 96.97 seconds against 105.52 seconds for PSO, while fixed-time control, GA and ACO recorded 127.13, 149.43 and 113.66 seconds respectively. Under high demand the gap widened dramatically: the proposed method held waiting time to 207.72 seconds, against 315.91 seconds for PSO, and it also posted the highest average vehicle speed at 7.92 meters per second. On the OSM network the framework’s waiting time fell to 33.07 seconds from PSO’s 48.88 seconds, and its standard deviation dropped from 17.42 seconds to 3.54 seconds, indicating far more consistent performance across repeated runs.

Statistical analysis supported these gains. On the OSM network, one-way ANOVA and the Kruskal-Wallis test both found significant differences among the five methods at p below 0.001, and Holm-corrected Welch tests against each benchmark yielded p-values of at most 1.17 times 10 to the power of -5, with Cohen’s d effect sizes ranging from 0.71 against ACO to 2.33 against fixed-time control. An ablation study underscored the value of the learning component: removing the PPO agent raised mean waiting time under high demand from 60.38 seconds to 64.91 seconds, a 6.98% improvement attributable to adaptive operator selection, while adding essentially no computational cost, with mean optimization time rising only from 43.74 to 43.79 seconds.

The authors also examined how the agent actually behaved. Across the high-traffic experiments on both networks, the learned policy used all six operators, with the elite-guided update chosen most often at 20.61% of selections and the restart strategy least often at 13.58%. The relatively balanced distribution suggests the agent learned to alternate between exploration-oriented moves such as novelty-guided perturbation, mutation and restart, which preserve swarm diversity, and exploitation-oriented moves such as elite guidance and local search, which sharpen promising timing plans. The improvement was notably larger on the OSM network than on the regular grid, which the authors attribute to the larger and more complex search space where adaptive strategy selection pays off most.

Important caveats remain. All traffic demand in the study was synthetically generated rather than drawn from field measurements, so although the OSM network captures realistic road geometry, the experiments do not reproduce observed real-world demand patterns, and validation on field data remains open. The framework also assumes reliable traffic state information, whereas practical deployment would face noisy sensors, communication delays and incomplete observations. The optimization objective centered on waiting time alone, leaving fuel consumption, emissions, pedestrian safety and emergency vehicle priority for future work. Even so, the study demonstrates that within simulated urban networks, reinforcement learning can serve effectively as an internal strategist for population-based optimizers, directly improving the quality of the traffic signal timing plans they produce.

Subject of Research: A reinforcement learning-guided multi-strategy particle swarm optimizer for adaptive traffic signal timing

Article Title: Proximal Policy Optimization-guided novelty-aware multi-strategy particle swarm optimization for adaptive traffic signal optimization

Article References: R., M., & J., A. P. (2026). Proximal Policy Optimization-guided novelty-aware multi-strategy particle swarm optimization for adaptive traffic signal optimization. Frontiers in Artificial Intelligence, 9, Article 1965356. https://doi.org/10.3389/frai.2026.1965356

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1965356

Keywords: traffic signal optimization, particle swarm optimization, Proximal Policy Optimization, reinforcement learning, SUMO, novelty search, swarm intelligence, OpenStreetMap, intelligent transportation systems, adaptive traffic control, Proximal, Policy