Automated Machine Learning System Identifies Malfunctions in Large Hadron Collider Detector

A newly published study on automated monitoring at the Large Hadron Collider reveals that combining advanced statistical tests with unsupervised machine learning enables rapid, objective identification of hardware malfunctions in particle physics experiments. Researchers operating the Compact Muon Solenoid detector introduced the AutoDQM platform, an automated data quality monitoring architecture designed to evaluate collision records without relying on continuous visual inspection. Tested on the full record of proton-proton collisions gathered by the CMS experiment throughout 2022, the framework deploys anomaly detection algorithms based on the beta-binomial probability distribution and principal component analysis. The combined analytical pipelines identified anomalous collision data affected by significant detector malfunction at an incidence rate four to six times greater than clean operational runs. Overall, the automated system successfully isolated over fifty percent of the year’s major operational data failures using monitoring plots from the Level-1 Trigger alone, while restricting false anomaly flags on certified good data to under fifteen percent.

The operational scope of the Compact Muon Solenoid experiment creates a massive monitoring burden that necessitates automated surveillance tools. Stationed at CERN’s Large Hadron Collider, the multipurpose apparatus gathers data from high-energy particle collisions to investigate fundamental physics following the Higgs boson discovery. To measure collision events, a global particle-flow algorithm integrates complementary measurements from concentric subdetectors arranged around the beam axis. Charged particle trajectories are tracked through a multi-layer silicon tracker within a 3.8-tesla magnetic field. Surrounding the tracker, an electromagnetic calorimeter absorbs electrons and photons, while an interleaved brass and plastic scintillator hadron calorimeter records energetic hadron showers. Beyond these calorimeters, layers of drift tubes, cathode strip chambers, resistive plate chambers, and gas electron multipliers monitor penetrating muons inside the flux-return yoke. While proton collisions exceed 30 megahertz, Level-1 Trigger hardware boards filter occurrences to at most 100 kilohertz in under four microseconds before computing farms store approximately 1 kilohertz on disk.

Managing the operational integrity of this complex instrumentation requires continuous surveillance of thousands of statistical distributions generated in real time. Standard data-taking runs, spanning several minutes to multiple hours, are converted into extensive collections of histograms representing detector occupancies, trigger metrics, and reconstruction parameters. Human operators known as shifters traditionally examine selected histograms through online and offline graphical interfaces, comparing current plots against reference curves from known stable runs. However, manual visual comparison across dozens or hundreds of complex distributions induces cognitive fatigue and invites error, while standard statistical tests such as Kolmogorov-Smirnov and basic chi-squared comparisons provide highly unreliable anomaly discrimination. Undetected hardware failures compromise downstream physics analyses, resulting in the annual loss of several percent of collected data, corresponding to dozens of hours of lost beamtime. AutoDQM was engineered as an automated web-based service capable of assessing online and offline histograms across subdetectors, thereby eliminating the subjective hazards of manual histogram inspection.

The mathematical foundation of AutoDQM begins with a statistical test based on the beta-binomial probability function, treating histogram bin contents as discrete outcomes. For any evaluated data histogram containing an integral total of D entries, the integer count d_i in bin i is modeled as an outcome drawn from D observations. Comparing against a reference histogram with integral R and bin entries r_i from previous stable runs, the likelihood L_i is calculated using a beta-binomial formulation with alpha set to 1 plus r_i and beta set to 1 plus R minus r_i. Evaluated across all bins via NumPy arrays and SciPy routines, each likelihood is compared against the maximum likelihood for that bin. The resulting relative likelihood ratio is transformed into a standardized pull value Z_i. To prevent false anomalies on large event samples, R and r_i are scaled by a factor tau, enforcing a minimum prediction uncertainty tolerance of approximately one percent.

To accommodate operational drift and systematic shifts in beam conditions, the beta-binomial method evaluates data against multiple historical reference runs simultaneously. Reconstructed particle distributions alter their geometric profiles in response to simultaneous collisions, known as pileup, causing normal distributions to shift between run intervals. AutoDQM resolves this by calculating relative likelihoods against several reference runs and deriving final pull values from their average. If an observed bin matches even a single reference run among a chosen set of eight, its calculated pull remains bounded, preventing changing beam conditions from generating artificial error flags. In an illustrative scenario where seven references yield near-zero likelihood but an eighth yields 0.33, the final pull value evaluates to approximately 2.5 rather than infinity. Across an entire distribution, AutoDQM computes the normalized mean chi-squared statistic alongside a modified maximum pull magnitude, which explicitly corrects the lowest single-bin likelihood for the look-elsewhere effect.

Complementing statistical tests, AutoDQM implements unsupervised machine learning architectures that model normal detector behavior without requiring prior examples of hardware breakdowns. Supervised learning algorithms are impractical for detector monitoring because severe hardware failures are exceptionally rare and past malfunctions do not reliably mirror future failure modes. The unsupervised pipeline utilizes principal component analysis from scikit-learn, training exclusively on a verified dataset of 216 flawless operational runs that each lasted at least thirty minutes. Input histograms are transformed into two key principal components that capture primary systematic variance across normal collision data. To prevent low-occupancy statistical noise from destabilizing the learned components, bins are merged iteratively during preprocessing until every bin accounts for at least 0.33 percent of total entries across the training set. When reconstructing distributions from the latent representation, a rectifier function forces negative bin values to zero, preventing unphysical histogram estimates while avoiding bias during inverse transformation.

A significant mathematical hurdle addressed during the development of the machine learning pipeline involved correcting an intrinsic bias associated with the duration of collision runs. Initial evaluations using the normalized sum of squared errors between original and reconstructed histograms revealed an inverse correlation with total histogram entries, systematically assigning inflated anomaly scores to shorter runs. To establish an unbiased metric, researchers adapted the beta-binomial formulation to evaluate machine learning reconstructions. The original unnormalized data histogram is compared against the unnormalized reconstructed distribution scaled by one hundred, suppressing reconstruction variance and isolating observational uncertainty. Because raw chi-squared values remained systematically lower for distributions with fewer entries, the metric was normalized by dividing the chi-squared score by the cube root of total entries D. This modified metric effectively eliminates sample-size dependence, providing an objective threshold where values exceeding 0.4 reliably signal significant deviations from normal operational behavior across both short and extended collision runs.

To validate AutoDQM against authentic detector conditions, the research team evaluated the tool using an independent benchmark comprising 308 collision runs from 2022, representing 36 inverse femtobarns of integrated luminosity. Operational quality determinations were established independently by the CMS Physics Performance and Datasets group without reference to AutoDQM. The assessment dataset included 265 good runs, defined as having at least ninety percent of data certified for physics, and 43 bad runs, defined as containing over fifty percent invalid data or more than 40 inverse picobarns of compromised data. Selected runs lasted at least five minutes and accumulated at least 3 inverse picobarns of data. Performance was measured across 62 distinct online monitoring histograms from the Level-1 Trigger, covering calorimeters and muon chambers. Runs categorized as defective solely due to silicon tracker issues were excluded, as Level-1 Trigger hardware does not receive silicon tracker information.

The quantitative evaluation demonstrated that both statistical and machine learning algorithms provide strong discrimination between healthy detector states and critical operational anomalies. Performance was measured using receiver operating characteristic curves based on the mean number of histogram flags per run alongside the fraction of runs exhibiting at least one, three, or five flagged distributions. In individual tests where anomaly thresholds were tuned so that good runs triggered fewer than three histogram flags on average, defective runs produced three to four times more flags. Similarly, with less than twelve percent of good runs flagged, the system detected between 35 and 50 percent of certified bad runs. The highest discrimination was achieved by deploying beta-binomial and principal component analysis tests simultaneously. Under this combined configuration, bad runs accumulated four to six times more anomaly flags than good runs. Over 55 percent of all bad runs triggered at least three separate anomaly flags at operating thresholds where only thirteen percent of certified good runs exceeded that flag count.

In practical deployment, AutoDQM presents its statistical evaluations through a web interface that highlights anomalous histogram regions, accelerating expert diagnostics during live operations. For two-dimensional distributions, pull values appear as heat maps displaying localized entry deficits in blue and unexpected excesses in red, while one-dimensional plots overlay measured data against probability-weighted reference averages alongside pull panels. The utility of this spatial mapping was demonstrated on CMS endcap cathode strip chambers, an array of 540 detectors where rare cluster failures occasionally disable dozens of chambers simultaneously. During run 356,001, AutoDQM flagged multiple cathode strip chamber histograms, highlighting temporary loss regions in blue while displaying chronically disabled chambers in white, enabling experts to separate transient failures from baseline conditions. With its source code published on GitHub, the automated platform establishes a foundation for expanding unsupervised machine learning surveillance across additional CMS subdetectors, safeguarding data integrity as collision volumes escalate during future Large Hadron Collider operations.

Subject of Research: Automated anomaly detection for particle detector data quality monitoring

Article Title: Anomaly Detection for Automated Data Quality Monitoring in the CMS Detector

Article References: Brinkerhoff, A., Sutantawibul, C., Suarez, I., White, R., Daumann, C., Guiang, J., Freer, C., May, S., Marsh, B., Acosta, D., Aubuchon, A., Barberis, E., Bundock, A., Campagnari, C., Collins, E., Epps, P., Erdmann, J., Flaecher, H., Huang, J., … the CMS Muon Detector Collaboration (2026). Anomaly Detection for Automated Data Quality Monitoring in the CMS Detector. EPJ Research Infrastructures, 10(1), Article 4. https://doi.org/10.1007/s41781-025-00147-2

Image Credits: AI Generated

DOI: 10.1007/s41781-025-00147-2

Keywords: Compact Muon Solenoid, Large Hadron Collider, Data Quality Monitoring, AutoDQM, Anomaly Detection, Machine Learning, Particle Physics, Principal Component Analysis, Anomaly, Detection, Automated, Data