Tracking cosmic ghosts

The idea was so far-fetched it seemed like science fiction: create an observatory out of a one cubic kilometer block of ice in Antarctica to track ghostly particles called neutrinos that pass through the Earth. But speaking to Benedickt Riedel, global computing manager at the IceCube Neutrino Observatory, it makes perfect sense.

“Constructing a comparable observatory anywhere else would be astronomically expensive,” Riedel explained. “Antarctica ice is a great optical material and allows us to sense neutrinos as nowhere else.”

Neutrinos are neutral subatomic particles with a mass close to zero that can pass through solid materials at near the speed of light, rarely reacting with normal matter. They were first detected in the 1950s in experiments that operated near nuclear reactors, which also generate these particles. They were further found to be created by cosmic rays interacting with our atmosphere. But astrophysicists believed they were likely widespread and caused by a variety of cosmic events, if only they could be detected.

Importantly, scientists believed they could be critical clues to other phenomenon. “20 percent of the potentially visible Universe is dark to us,” Riedel explained. “That’s mostly because of distances and the age of the Universe. High energy light is also hidden. It is absorbed or undergoes transformation that makes it hard to trace back to a source. IceCube reveals a slice of Universe we haven’t yet observed.”

An Important New Tool in the Multi-Messenger Astronomy Toolbox

Multi-messenger astronomy describes an approach that combines observations of light, gravitational waves, and particles to understand some of the most extreme events in the Universe. Neutrinos play an important part in this type of research.

Prior to 1987, with the explosion of Supernova 1987a, all extra-solar astronomical observations were photon-based. Today, additional detection systems add to our view of the cosmos, including all sky surveys and gravitational wave detectors. However, most observatories can only look at a small portion of the sky. IceCube, because of the nature of neutrinos, can observe these particles’ flights from any direction, and therefore act as a full-sky sentinel.

The block of ice at the Amundsen-Scott South Pole Station in Antarctica — up to a hundred thousand years-old and extremely clear — is instrumented with sensors between 1,450 and 2,450 meters below the surface. As neutrinos pass through the ice, they may interact with a proton or neutron, producing photons which then travel through the ice, and can be detected by a sensor. The sensors transform these signals from neutrino interactions — a handful an hour — into digital data that is then analyzed to determine whether they represent a local source (Earth’s atmosphere) or a distant one.

“Based on the analysis, researchers are also able to determine where in the sky the particle came from, its energy, and sometimes, what type of neutrino — electron, muon or tau — it was,” said James Madson, executive director at the Wisconsin IceCube Particle Astrophysics Center.

In 2017, IceCube detected a neutrino with an energy of 290 teraelectronvolts (TeV) and sent out an alert. The detection triggered an extensive campaign involving more than twenty space- and ground-based telescopes. They identified a blazar 3.5 billion light years away, identifying a high energy cosmic ray source for the first time and launching a new era in multi-messenger detection, according to Riedl.

“We continuously search our dataset in near-real time for interesting neutrino events,” he explained. “We found one and sent out an email alert to the community. They followed up with all these other electromagnetic observations, pinpointing a known gamma ray source. They also found, over the course of a month, an increased activity from the source.”

IceCube Discovers Evidence of High-energy Electron Antineutrino

On March 10, 2021, IceCube announced the detection of a Glashow resonance event, a phenomenon predicted by Nobel laureate physicist Sheldon Glashow in 1960. The Glashow resonance describes the formation of a W? boson — an elementary particle that mediates the weak force — during the interaction of a high-energy electron antineutrino with an electron, peaking at an antineutrino energy of 6.3 petaelectronvolts (PeV). Its existence is a key prediction of the Standard Model of particle physics. The results further demonstrated the ability of IceCube to do fundamental physics. The result was published on March 10 in Nature.

While this energy scale is out of reach for current and future planned particle accelerators, natural astrophysical phenomena are expected to produce antineutrinos that reach beyond PeV energies. The news of the Glashow resonance discovery, “suggests the presence of electron antineutrinos in the astrophysical flux, while also providing further validation of the standard model of particle physics,” the authors wrote. “Its unique signature indicates a method of distinguishing neutrinos from antineutrinos, thus providing a way to identify astronomical accelerators that produce neutrinos via hadronuclear or photohadronic interactions, with or without strong magnetic fields.”

Neutrino detections require significant computing resources to model the detector behavior and differentiate extra-solar signals from background events created from cosmic ray interactions in the atmosphere. Riedel serves as the coordinator for a large community of researchers — as many as 300 by his estimates — who use the Frontera supercomputer at the Texas Advanced Computing Center (TACC), a National Science Foundation (NSF)-funded resource for the national community.

IceCube was awarded time on Frontera as part of the Large Scale Community Partnership track, which provides extended allocations of up to three years to support long-lived science experiments. IceCube – which has collected data for 14 years and was recently awarded a grant from NSF to expand operations over the next the next few years — is a premier example of such an experiment.

“Part of the resources from Frontera contributed to that discovery,” Riedl said. “There’s years of Monte Carlo simulations that went into it to figuring out that we could do this.”

IceCube uses computing resources from a number of sources, including the Open Science Grid, the Extreme Science and Engineering Discovery Environment (XSEDE), their own local supercomputing cluster, and recently the Amazon Web Services cloud. Frontera is the largest system utilized, however, and can handle a large part of the computational needs of the neutrino community, reserving local or cloud resources for urgent analyses, Riedel says.

“A lot of the computing on Frontera may not be directly associated with discoveries, but it helps down the road, to discern signals better and develop new algorithms,” he said.

Modeling Ice and Following Up on Promising Signals

The projects that IceCube scientists use Frontera for vary, but they typically either involve calculations to better understand the optical nature of the ice generally (so the trajectory and other characteristics of neutrino detections can be accurately determined); or computations to analyze specific events that are deemed significant.

The first type of computation uses primarily ray tracing to calculate the path of the light in the ice from high-energy electrically charged particles produced when neutrinos interact. The rays can scatter or be adsorbed by defects in the ice, complicating analysis. Using graphics processing units (GPUs), Riedel has found, can speed up the simulations to studying light the light propagation in the ice by hundreds of times. The IceCube team is among the largest users of the Frontera GPU subsystem that includes NVIDIA RTX GPUs.

The second type of computation occurs when scientists receive an alert that says they have received an interesting signal. “We kick off a calculation to analyze the event that can scale to one million CPUs,” Riedl said. “We don’t have those, so Frontera can give us a portion of that computational power to run a reconstruction or extraction algorithm. We get those type of events about once a month.”

“Large scale simulations of the IceCube facility and the data it creates allow us to rapidly and accurately determine the properties of these neutrinos, which in turn exposes the physics of the most energetic events in the universe,” said Niall Gaffney, TACC Director of Data Intensive Computing. “This is key to validating the fundamental quantum-mechanical physics in environments that cannot be practically replicated on earth.”

Today’s astronomers can observe the universe in many different ways, and computing is now central to almost all of them. “We’ve moved from the traditional view of a guy with a telescope looking up at the sky, to large scale instruments, to now particle physics and particle observatories,” Riedl said. “With this new paradigm, we need large amounts of computing for short periods of time to do big time sensitive computing, and big scientific computing centers like TACC help us do our science.”

###