Resilient execution of distributed X-ray image analysis workflows

At synchrotron radiation facilities, X-ray tomography experiments generate torrents of projection data that must be reconstructed into detailed 3D images, often in near real time. A team at Argonne National Laboratory and the University of Chicago has now shown how to make these distributed image analysis workflows dramatically more resilient to the failures that plague large-scale computing systems. Their new study, published in Frontiers in High Performance Computing, introduces checkpointing and load redistribution techniques that reduce the impact of failures by up to 500 times while adding less than 3 percent overhead to the reconstruction process.

The challenge stems from the sheer scale of modern tomographic reconstruction. Iterative reconstruction methods, which refine image estimates by comparing them against actual detector measurements over many iterations, are computationally intensive but produce high-quality results from limited or noisy data. To speed things up, researchers partition the reconstructed image and raw data into slices that can be processed independently in parallel across many compute nodes. This embarrassingly parallel structure allows performance to scale with the number of available nodes, but it also means the workflow runs on high-performance computing systems with thousands of components, including CPUs, GPUs, memory, communication links, and storage systems, all of which are susceptible to failures ranging from software crashes to complete system outages.

When a failure occurs, the progress of the affected task is typically lost unless appropriate recovery mechanisms are in place. Restarting failed tasks from scratch can recover the image but at a high cost, potentially doubling runtimes and leaving many nodes idle for extended periods. In extreme cases, the failure rate may surpass the system’s ability to make progress, trapping the workflow in an endless cycle of recomputation. The team’s performance model demonstrates that frequent failures can increase reconstruction time by up to four orders of magnitude without proper resilience mechanisms. For instance, if the mean time to failure of a single task is approximately one week, a reconstruction pipeline with 1,000 tasks would, on average, fail in about 10 minutes, often shorter than the runtime of many workflows of this size.

Current fault tolerance mechanisms typically rely on checkpoint-restart techniques, which periodically save the state of each process so failed tasks can be restarted from the last checkpoint. While this reduces lost computation, it introduces I/O overhead due to frequent writes to parallel file systems, which can become a bottleneck in high-concurrency environments. Advanced systems such as VeloC and SCR alleviate this by asynchronously writing checkpoint data to local storage and deferring parallel file system writes. However, these systems are optimized for tightly coupled parallel tasks that require all tasks to restart simultaneously in the event of a failure. Tomographic reconstruction, in contrast, is embarrassingly parallel, and the failure of a single task does not require the restart of surviving tasks that have no dependencies on the failed ones.

The Argonne team’s solution incorporates two key innovations. First, they introduce an asynchronous, non-blocking checkpointing mechanism that operates independently for each task and supports variable-sized checkpoints reflecting the evolving critical state required for recovery. Tasks can continue without being blocked while critical state data is saved progressively, ensuring minimal disruption during execution. Each checkpoint is self-contained, embedding all necessary information to restore progress, such as completed iterations and sinogram indices, which eliminates computation and synchronization overhead during failure recovery. The team also dynamically adjusts the checkpoint period using a modified Young/Daly formula that accounts for changes in task availability and I/O performance, reconfiguring whenever a failure occurs.

Second, in the event of a failure, the team employs a lazy, balanced redistribution of failed tasks. Surviving tasks are allowed to continue their processing without interruption while the checkpointed slices of the failed tasks are redistributed among the survivors in the background. Rather than immediately recovering lost progress from checkpoints, surviving tasks treat the checkpointed progress as the most recent state of the failed task, and recovery is embedded into future reconstruction iterations. This avoids resource locking and enables tasks to utilize their resources more efficiently for ongoing computations. The redistribution strategy minimizes expensive read overheads associated with parallel file systems by relying on direct communication between surviving tasks, ensuring efficient resource utilization and balanced load distribution across available tasks.

To evaluate their effectiveness, the team implemented a 3D tomographic reconstruction pipeline based on the Trace reconstruction engine, supporting three iterative algorithms: Algebraic Reconstruction Technique, Simultaneous Iterative Reconstruction Technique, and Maximum Likelihood Expectation Maximization. They deployed it across Argonne’s Polaris supercomputer, where each compute node features a single AMD EPYC Milan processor with 64 physical cores and 512 GB of memory, interconnected by a 200 Gbps high-speed network. Checkpoints were managed by VeloC operating in asynchronous mode and stored in a Lustre file system. The team used two sample datasets from TomoBank, one comprising 1,500 X-ray projections of borosilicate glass spheres and another corresponding to a portion of an electronic circuit with 1,204 projections.

The results demonstrate that the proposed resilience techniques significantly reduce failure impact. Without resilient solutions, reconstruction time increases exponentially as the mean time to failure decreases, surging from 60 seconds in a failure-free environment to over 100,000 seconds at a mean time to failure of 10 seconds, a four-order-of-magnitude slowdown. Introducing checkpointing dramatically reduces this overhead by preserving progress and avoiding full recomputation of lost work, reducing reconstruction time to under 1,000 seconds at the same failure rate, a 100-fold improvement. Adding dynamic redistribution with balance-aware assignment and lazy recovery further enhances performance by an additional 4.3 times. Under the extreme condition of a mean time to failure of 10 seconds, the fully resilient pipeline completes the reconstruction in just 184 seconds, only 3 times slower than ideal execution and over 500 times faster than the naive resilient baseline.

These remarkable improvements come with minimal overhead. The combined checkpointing and communication overhead for dynamic redistribution remains consistently below 3 percent of the ideal reconstruction time across all failure scenarios. The team also evaluated robustness across diverse conditions, varying sinogram sizes, datasets, and reconstruction algorithms. Changes in sinogram size had minimal impact on efficiency, and as sinogram size increased from 640 by 640 to 1,280 by 1,280 pixels, the overhead ratio decreased by approximately 3.5 times because larger sinograms require longer computation, allowing the asynchronous, non-blocking mechanisms to overlap checkpointing with computation more effectively. The solutions also generalize across parallel-beam and cone-beam geometries, though cone-beam setups with cross-sinogram dependencies can only apply checkpointing without compromising algorithm correctness.

The team notes that their resilient solutions rely on two key properties common to many reconstruction pipelines: the iterative reconstruction pattern, in which the reconstructed object represents an intermediate state that can serve as a progress checkpoint, and partial updates with data-parallelism, where data can be partitioned and processed by independent tasks. Any tomographic reconstruction workflow exhibiting one or both of these properties can leverage these techniques. While the current implementation is CPU-based, the design is independent of any specific hardware architecture, and the approach can be adapted for GPU-based reconstruction pipelines with only minor modifications. Future work will address data transmission losses, file corruptions, storage failures, and the growing demand for real-time reconstruction with strict timing requirements.

Subject of Research: High Performance Computing

Article Title: Resilient execution of distributed X-ray image analysis workflows

Article References: Nguyen, H. D., Bicer, T., Nicolae, B., Kettimuthu, R., Huerta, E. A., & Foster, I. T. (2025). Resilient execution of distributed X-ray image analysis workflows. Frontiers in High Performance Computing, 3, Article 1550855. https://doi.org/10.3389/fhpcp.2025.1550855

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1550855

Keywords: X-ray tomography, checkpointing, fault tolerance, distributed computing, synchrotron, reconstruction workflows, Resilient, execution, distributed, X-ray, image, analysis