White Paper Defines Infrastructure Needs for HL-LHC Analysis

A comprehensive white paper published in Computing and Software for Big Science outlines the critical infrastructure requirements for high-energy physics (HEP) analysis facilities as the field prepares for the High-Luminosity Large Hadron Collider (HL-LHC) era. The document, authored by a large consortium of researchers including D. Ciangottini, A. C. Forti, and L. Heinrich, synthesizes the current status of research and development (R&D) efforts aimed at evolving the computational infrastructure necessary to handle the anticipated surge in data volume. The HL-LHC is scheduled to begin operations in 2029, a period during which analysts may need to process up to an order of magnitude more data than is currently handled, necessitating a fundamental shift in how analysis facilities are designed and operated.

The white paper defines an Analysis Facility (AF) as a set of infrastructure and services that provide integrated data, software, and computational resources to execute elements of an analysis workflow. These resources are shared among members of a Virtual Organization (VO) and are supported by the hosting organization. The authors emphasize that the distinction between AFs and the existing Worldwide LHC Computing Grid (WLCG) is becoming increasingly blurred, suggesting that the integration of AFs within the existing WLCG infrastructure is vital for ensuring broad accessibility and a standardized user experience. This integration aims to facilitate smooth data access, dynamic allocation of computing resources, and easier support from various sites, moving away from the notion of AFs as isolated entities.

A central theme of the document is the evolution of user requirements, which must drive the R&D agenda. The authors identify the ability to perform fast research iterations on large datasets interactively as a primary need. In the current landscape, this is often satisfied by classic facilities providing interactive single login nodes or small single-node Jupyter notebooks. However, as dataset sizes increase, this model is becoming insufficient. The new requirement is for a distributed and interactive analysis style, where a user can coordinate work in an interactive session while dynamically connecting to a scale-out system that provides additional resources. The expectation is that processing on these additional resources starts in parallel almost immediately, with only a short lag time due to resource provisioning, allowing data to be aggregated and sent back to the user in real-time.

The transition from the research and development phase to the final measurement phase presents another significant challenge. As an analysis matures, the usage pattern shifts from rapid iteration to a more controlled, batch-schedulable mode. Users in this phase may not require full interactivity and may prefer to prepare analysis configurations ahead of time for asynchronous scheduling. The white paper highlights the importance of easily converting workloads from an interactive setup into a headless mode amenable to batch systems. This is particularly relevant for Jupyter notebooks, which must be re-configurable from an interactive, dynamically scaling environment into a non-interactive workload that requests a fixed number of workers and scales down reliably to free resources.

Machine learning (ML) is identified as a major driver of infrastructure change, affecting data-taking, simulation, reconstruction, and analysis across the board. ML methods enable significant acceleration due to the high degree of parallelization possible in neural networks, which maps well to hardware such as GPUs. Consequently, AFs must provide an efficient Python-based analysis environment and access to heterogeneous hardware both interactively and in batch mode. The dominance of Python in ML research, with tools like PyTorch, SciKit-Learn, and TensorFlow, dictates the need for robust support for these ecosystems. Users expect to be able to start interactive sessions with GPU access, and for large-scale model training, which can exceed 24 hours, access to batch GPUs via standard workload schedulers is increasingly required.

Data organization, management, and access (DOMA) are critical components of the AF infrastructure. The paper discusses various models for data access, including hosting data at CERN, creating other facilities with large storage similar to CERN, or using remote access with latency hidden by caches. Caching is particularly beneficial because analysis workflows frequently access the same data repeatedly. The authors note that XRootD-based XCaches have been used in testing, offering the advantage that applications can read only the required portions of files, reducing transfer volumes. The choice between POSIX file systems and object stores is also debated, with users favoring POSIX for familiarity, but object stores offering scalability and efficiency for large-scale data serving. The paper suggests that a general need exists for both static files and object stores to be accepted as input and output formats, with the most efficient formats preferred for the largest-scale processing tasks.

Federated Identity Management (FIM) and Authentication, Authorization, and Infrastructure (AAI) are essential for equitable access to resources in a VO-wide manner. The current WLCG FIM model relies on x509 certificates, which are difficult to maintain and incompatible with cloud services. The WLCG is working towards an AAI based on OAuth2.0 and tokens, which will replace x509 certificates by the HL-LHC era. This new token infrastructure is fully compatible with cloud and web services such as JupyterHub and Kubernetes, promising a greatly improved user experience. The integration of analysis infrastructure services with this new AAI is highly desirable to allow for uniform access to all resources for authorized users, while still allowing VOs to decide on specific authorization modes.

The paper also addresses the growing role of accelerator resources, particularly GPUs. It is expected that both the availability and demand for GPU resources will increase, driven by the needs of machine learning and HPC facilities. AFs must provide software hooks to integrate this hardware, as the variety of GPU models and the complex memory management they require pose new challenges. Currently, GPUs are often used on dedicated resources or as single interactive GPUs, but the paper discusses the potential for time-sharing and physical partitioning in newer models. Some AFs are deploying specialized software, such as NVIDIA’s Triton inference servers, for inference-as-a-service, which can be used from local batch jobs or interactive nodes. The integration of commercial cloud resources for GPU access is also being investigated, with models that allow on-demand allocation of resources.

Analysis portability and preservation are core requirements for AFs, ensuring that users can realize their desired software stack and preserve their analyses for future reuse. The paper discusses several methods for building software stacks, including CVMFS, Conda, and Linux containers. CVMFS is the dominant solution for software distribution in HEP, but it has disadvantages regarding preservation and runtime access. Conda is a cross-platform package manager that solves portability issues but requires locking to specific versions for true preservation. Linux containers, in the OCI format, allow users to assemble a self-contained, preservable software stack. The paper notes that these methods can be combined, as seen in LHCb’s use of Conda with CVMFS, and that the construction of well-maintained software stacks may require expertise from AF and experiment experts. Tools like repo2docker and Binder are mentioned as ways to abstract the image building process away from the user.

Monitoring and metrics are crucial for guiding infrastructure development and allowing users to make informed decisions about which infrastructure to use. The paper categorizes metrics into four areas: user experience, user trend, performance, and facility metrics. User experience metrics include support response times and user satisfaction surveys, while user trend metrics can inform future development by tracking data sources, input formats, and resource usage. Performance metrics can be assessed using benchmarks like the IRIS–HEP Analysis Grand Challenge, which provides complete analysis workflows for testing functionality and performance. Facility metrics quantify technical performance, such as resource idle time and job failure rates. The paper also highlights the growing importance of sustainability metrics, such as the carbon footprint per user/job, which will play a significant role in evaluating facilities’ viability in the future.

The white paper concludes by emphasizing the need for cross-experiment working groups to follow up on the identified goals. These groups should focus on collecting analysis workflow use-cases, developing tools for monitoring user jobs, and discussing the integration of object stores into the experiments. The authors stress that the goals outlined in the paper should guide the development of future infrastructures to enable the full scientific potential of HEP experiments in the coming decades. The document serves as a roadmap for the community, identifying key open questions and areas for further R&D, ensuring that the analysis infrastructure evolves in a way that supports the complex and data-intensive nature of modern high-energy physics research.

Subject of Research: Infrastructure requirements for high-energy physics analysis facilities for the HL-LHC

Article Title: Analysis Facilities for the HL-LHC White Paper

Article References: Ciangottini, D., C. Forti, A., Heinrich, L., Skidmore, N., Alpigiani, C., Aly, M., Benjamin, D., Bockelman, B., Bryant, L., Catmore, J., D’Alfonso, M., Delgado Peris, A., Doglioni, C., Duckeck, G., Elmer, P., Eschle, J., Feickert, M., Frost, J., Gardner, R., … Watts, G. (2025). Analysis Facilities for the HL-LHC White Paper. Computing and Software for Big Science, 9(1), Article 14. https://doi.org/10.1007/s41781-025-00133-8

Image Credits: AI Generated

DOI: 10.1007/s41781-025-00133-8

Keywords: High Energy Physics, HL-LHC, Analysis Facilities, Computing Infrastructure, Machine Learning, Data Management, GPU Resources, Software Preservation, WLCG, HEP Software, Analysis, Facilities