Graph Neural Network Predicts Job Wait Times Across Huge Computing Grids

One of the busiest distributed computing systems in science can now forecast how long an individual job will take to complete with striking accuracy, thanks to a graph neural network trained on millions of real production records. Researchers working with the PanDA workload management system, which coordinates computing for the ATLAS experiment at the Large Hadron Collider and the ePIC collaboration at the Electron-Ion Collider, report that their model, PanDA-GNN, reached a coefficient of determination of 0.94 on a held-out test set, with a mean absolute error of 84.72 minutes and a median absolute error of just 39.62 minutes. The work, published in EPJ Research Infrastructures, shows that representing jobs as nodes in a graph, connected to temporally earlier jobs sharing the same site, processing type and resource type, captures operational information that flat, per-job feature vectors miss entirely.

Turnaround time, in PanDA’s data model, is the full interval from job submission to output delivery, combining queueing delay with execution time. It is the quantity users actually experience: how long after hitting submit do results arrive. Predicting it is far harder than predicting execution time alone, because the two components obey different dynamics. Execution time depends mostly on the job itself and the resources it lands on. Queueing delay, by contrast, is driven by workload pressure, pilot availability, priority and site-specific provisioning behavior, all of which shift hour by hour across facilities that process millions of jobs per week. The authors treat the problem as relational rather than purely tabular, arguing that jobs sharing a site, resource type or processing context are not independent samples; their outcomes reflect common queue conditions and resource contention.

The dataset draws on PanDA job records from five of the most active computing sites, collected between November 2025 and January 2026. Jobs with turnaround times exceeding 2000 minutes were excluded to focus on the dominant operating regime, a filter that removed roughly 10.8 percent of jobs belonging to the most extreme long-turnaround tail. After filtering, the data were partitioned chronologically into training, validation and test sets to avoid temporal leakage: about 2.1 million jobs for training, roughly 0.46 million for validation and about 0.40 million for testing. The regression target was log-transformed as log(1 + T), where T is turnaround time in minutes, a standard move that tames the heavy right tail characteristic of distributed workloads. All metrics were converted back to minutes for interpretability.

Graph construction is deliberately local and causally strict. Jobs within each data split are ordered by start time and partitioned into contiguous chunks of 2000 jobs, a size chosen after testing alternatives of 1000 and 4000 with broadly comparable results. Each chunk becomes a directed subgraph in which edges run only from earlier jobs to later jobs. For each target job, up to 16 temporally preceding jobs sharing the same computing site, processing type and resource type are connected to it, subject to a maximum time gap of 24 hours. Each edge carries two temporal features, a log-scaled time difference and an exponential decay term. The neighborhood size mattered: relative to connecting only two past neighbors, using 16 improved test R-squared by about 4.0 percent and cut log-space RMSE and minute-scale MAE by roughly 22.7 percent and 36.3 percent respectively.

The architecture itself is a node-level graph transformer. Numerical inputs are standardized, categorical variables such as data type, project, processing mode, working group and resource category are encoded as learned embeddings, and a sinusoidal temporal encoding derived from relative job start time marks coarse chronological position. The concatenated node representation is projected into a shared hidden space and passed through a stack of edge-aware graph transformer blocks, each applying attention-based message passing with edge attributes, residual connections, layer normalization and a feed-forward transformation. On top of the shared encoder sit separate prediction heads: the primary head predicts log-transformed turnaround time, while auxiliary heads predict log-transformed runtime, log-transformed queue delay and a binary indicator for long-duration jobs. Training combines a weighted Huber and quantile loss for the main target, with auxiliary losses at fixed weights, plus sample reweighting and weighted subgraph sampling to improve performance on the long-duration tail. Optimization uses AdamW with a warm-up and cosine decay schedule, and early stopping selects the best checkpoint by validation R-squared.

The comparison against non-graph baselines is where the graph advantage becomes unmistakable. All four baselines, linear regression, a multilayer perceptron, XGBoost and LightGBM, were trained on the same chronological splits with the same input features. Linear regression performed worst, with an R-squared of 0.65 and a mean absolute error of 401.77 minutes, confirming the relationship between submission-time descriptors and turnaround is strongly non-linear. XGBoost, the strongest competitor, reached R-squared 0.92 with MAE of 163.10 minutes and MedAE of 76.36 minutes, while LightGBM also clearly beat linear regression and the MLP. PanDA-GNN then pushed R-squared to 0.94, cut MAE to 84.72 minutes and MedAE to 39.62 minutes. Relative to XGBoost, that corresponds to reductions of 48.12 percent in median absolute error, 48.05 percent in mean absolute error and 46.93 percent in RMSE, which reached 163.59 minutes, with mean absolute percentage error at 20.39 percent.

An edge ablation isolates what the connectivity itself contributes. Removing edges from the full model reduced test R-squared by about 4.8 percent and increased log-space RMSE and minute-scale MAE by roughly 28.9 percent and 47.7 percent respectively, with similar degradation on the validation split. Explicit inter-job links, in other words, carry predictive information beyond node features alone. Site-wise analysis reinforces the point: PanDA-GNN achieved higher R-squared and lower MAE and MedAE than XGBoost at every one of the five sites, indicating the gain is not driven by a single dominant facility but holds consistently across heterogeneous operating conditions. Diagnostic plots show residuals in log space centered near zero, with a mean residual of minus 0.05 and a standard deviation of 0.30, and absolute error percentiles of 39.62 minutes at the 50th percentile, 194.37 minutes at the 90th and 335.26 minutes at the 95th.

That percentile spread points to the model’s main weakness, which the authors confront directly. The gap between median and mean error signals a non-negligible long-error tail: a relatively small subset of long-running jobs contributes disproportionately to overall error, most likely because these jobs are more sensitive to transient queue conditions and system-level effects that cannot be inferred from submission-time metadata alone. The predicted distribution otherwise tracks the true turnaround distribution closely across the main mass of the data, with the largest deviations confined to the sparsely populated long-duration tail. Operationally, the authors note, scheduling decisions typically depend less on minute-level precision than on reliably distinguishing shorter jobs from substantially longer ones, which is exactly the regime where the model performs well.

The practical payoff lies in what accurate turnaround estimates would let operators do. PanDA already uses Scout Jobs, running a small subset of a task first to estimate resource requirements before releasing the rest, an effective but latency-adding two-stage process. Prior machine-learning work showed that task-level resource classes can be predicted at submission time, reducing reliance on Scout Jobs, but continuous job-level turnaround prediction under changing site conditions had not been addressed. The authors argue that reliable estimates could support pilot provisioning, queue-aware placement, backfilling and workflow-latency reduction, decisions that balance throughput and resource utilization for operators while giving users timelier, more predictable access to computing. PanDA’s scale makes this consequential: the system handles on the order of 700,000 concurrently running jobs and more than 6 million jobs per week across grid, cloud and high-performance computing resources.

Significant caveats remain before any production deployment. The graph is built offline over fixed chunks of jobs sorted by submission order; a live system would need incremental graph construction over a rolling stream of incoming jobs. The model produces point estimates only, whereas operational schedulers would benefit from calibrated uncertainty estimates or prediction intervals allowing conservative handling of low-confidence predictions, for example through wider resource buffers or routing uncertain jobs to sites with greater slack. The study is also restricted to the five highest-throughput PanDA sites over a fixed historical period, so the results should not yet be assumed to generalize to the full PanDA federation; extending to lower-volume sites, more heterogeneous workload profiles and previously unseen categorical values is future work. The underlying challenge, however, is not unique to PanDA. It is characteristic of large-scale distributed scientific computing everywhere, where heterogeneous resources, queueing dynamics and competing workloads shape the latency every user feels. The authors suggest the graph construction strategy itself could transfer to other operational tasks such as failure prediction, anomaly detection and resource-usage forecasting, positioning the work as a broader framework for infrastructure-aware workload modeling rather than a single-purpose predictor.

Subject of Research: Machine learning prediction of job turnaround time in the PanDA distributed computing workload management system

Article Title: Predicting Job Turnaround Time in Large-Scale Distributed Computing Environments with Graph Neural Networks

Article References: Chowdhury, T., Maeno, T., Akman, F. F., Boudreau, J., Dutta, S., Feng, S., Hoisie, A., Hsu, K.-C., Khan, R., Kim, J., Kilic, O. O., Klasky, S., Korchuganova, T., Lo, K., Outschoorn, V. I. M., Nilsson, P., Park, D. K., Podhorszki, N., Ren, Y., … Klimentov, A. (2026). Predicting Job Turnaround Time in Large-Scale Distributed Computing Environments with Graph Neural Networks. EPJ Research Infrastructures, 10(1), Article 14. https://doi.org/10.1007/s41781-026-00167-6

Image Credits: AI Generated

DOI: 10.1007/s41781-026-00167-6

Keywords: graph neural networks, PanDA, turnaround time prediction, distributed computing, ATLAS, job scheduling, workload management, queue delay, XGBoost, high-performance computing, research infrastructure, machine learning