Sliding-Window UCB Outperforms Competing Bandit Algorithms Under Simulated User Preference Drift

In a synthetic benchmark evaluating adaptive algorithms for dynamic recommendation systems, Sliding-Window Upper Confidence Bound consistently achieved the highest long-run reward when handling drifting user preferences. The findings, published in Neural Computing and Applications, evaluate how recommendation architectures perform when user interests evolve rather than remaining static over time.

To evaluate these dynamics, researchers Hema Chandra Puchakayala, Tyler Wallett, and Amir Jafari established a reproducible simulation framework. The framework models shifting user tastes as an evolving latent preference vector over content categories within a non-stationary multi-armed bandit environment, contrasting with conventional algorithms that assume stationary reward distributions.

Within this simulation architecture, the team implemented and compared four adaptive algorithms: Non-Stationary ε-Greedy, Sliding-Window UCB, CUSUM-UCB, and EXP3.S. Each method was tested on synthetically generated drifting-user data, measuring performance via average and cumulative rewards across 50 runs alongside confidence intervals.

The evaluation demonstrated that Sliding-Window UCB delivered the highest sustained performance, followed by CUSUM-UCB and Non-Stationary ε-Greedy, whereas the adversarially robust EXP3.S underperformed in this stochastic drift setting. The researchers concluded that methods explicitly discounting or resetting outdated observations are well suited to gradual user drift, noting that adversarial algorithms can be overly conservative when non-stationarity stems from user behavior rather than adversarial manipulation.

Recommendation systems operating in live, dynamic environments must routinely adapt to users whose preferences drift over time. However, many deployed bandit formulations continue to operate under the assumption of stationary reward distributions, creating a significant limitation when user habits change.

Subject of Research: Performance of adaptive multi-armed bandit algorithms under simulated user preference drift

Article Title: Benchmark of non-stationary bandit algorithms

Article References: Puchakayala, H. C., Wallett, T., & Jafari, A. (2026). Benchmark of non-stationary bandit algorithms. Neural Computing and Applications, 38(21), Article 886. https://doi.org/10.1007/s00521-026-12583-1

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12583-1

Keywords: non-stationary bandits, multi-armed bandits, recommendation systems, user preference drift, Sliding-Window UCB, CUSUM-UCB, EXP3.S, Non-Stationary epsilon-Greedy, latent preference vector, simulation framework, stochastic drift, reward distributions, cumulative reward, average reward, adaptive algorithms, adversarial bandits, content categories