Foundation Models Bring AI Speed to Supercomputer Simulations

Some of the world’s most demanding scientific calculations—climate models that stretch across kilometers of atmosphere, plasma simulations for fusion reactors, fluid dynamics for aircraft design—have long depended on numerical solvers that can take days, weeks, or even months on advanced supercomputers. A new perspective article argues that a different computational paradigm is now mature enough to change that equation: foundation models, the large pre-trained AI systems that have transformed language and vision, are being retooled to emulate the slow, iterative mathematics of physics simulation itself. The authors, a team from Oak Ridge National Laboratory, the University of Minnesota, and Saint Louis University, lay out a blueprint for building these models at supercomputing scale while keeping their predictions trustworthy enough for real science.

The core idea is deceptively simple. Instead of solving the partial differential equations that govern fluid flow, heat transfer, or atmospheric dynamics from scratch every time, a foundation model is trained on vast libraries of existing simulation data. Once trained, it can produce predictions in minutes that would otherwise demand days of number-crunching. The authors describe these systems as reusable base models: rather than starting over for each new application, scientists can adapt a pre-trained model to tasks such as inverse design—working backward from a desired outcome to find the parameters that produce it—or uncertainty quantification, without retraining from zero. The result is a workflow in which rapid, physics-informed optimization cycles become routine rather than exceptional.

The article is careful to position foundation models as complements rather than replacements for traditional simulation. The authors draw a direct line to older engineering practice: reduced-order and surrogate models, such as Reynolds-averaged approaches in fluid dynamics, have long been used to trade some physical fidelity for enormous gains in speed. Foundation models share that motivation but bring far greater expressive power, learning from large collections of simulation and experimental data using architectures like transformers and graph neural networks. The authors emphasize that these systems support high-throughput virtual experimentation and help researchers navigate design tradeoffs, while expert judgment and full-fidelity solvers remain essential anchors of the workflow.

The evidence for this approach spans multiple domains. In Earth system science, models such as DeepMind’s GraphCast and NVIDIA’s FourCastNet already use large spatiotemporal datasets to improve weather and climate prediction while providing actionable guidance to human forecasters. In materials science, models trained on crystal structure databases help researchers prioritize promising candidates for further study. Drug discovery efforts use similar techniques for protein structure prediction and molecular design. In fusion energy research, data-driven models assist physicists with plasma control and scenario planning in operational environments that are too complex for intuition alone. The common thread is that foundation models learn intricate physical and chemical relationships from diverse data, enabling rapid prediction across a broad range of physical domains.

One of the article’s most technically grounded sections concerns data management, an often-overlooked foundation of the entire enterprise. The authors argue that as foundation models scale to production on high-performance computing systems, the quality and organization of training data becomes the decisive factor in achieving both speed and generalization. Building high-quality datasets requires programmatic pipelines for collection, quality control, de-duplication, consistent labeling, and machine-readable provenance. They advocate moving beyond generic FAIR data principles toward a campaign-oriented governance model, in which simulation campaigns are recorded with standardized workflows and ontologies, and data curation is driven by impact metrics such as time-to-insight and downstream model quality rather than raw storage volume.

At exascale, the authors note, storing every high-order field produced by a simulation is simply infeasible, making compression a first-class design choice rather than an afterthought. Modern error-bounded lossy compression methods can dramatically reduce data footprints, but the authors insist that fidelity must be defined by physics-aware criteria—controlling errors in energy or gradients, for example—and by quantifying how compression perturbations affect downstream surrogates and emulators. Integrated in situ or in transit within the simulation pipeline, compression can even act as a learnable encoder, yielding compact features that minimize input-output and memory demands during pre-training and fine-tuning while maintaining strict error controls for diagnostics.

To make scientific data usable across domains, the authors propose a token-centric view of AI-readiness that mirrors how foundation models themselves process information. Descriptor tokens would capture meshes, coordinates, units, and provenance, while feature tokens represent compressed field content. Fusion mechanisms would bind these tokens across simulations, experiments, and modalities into coherent scientific narratives. Anchoring such tokens in community standards—netCDF-like schemas for Earth system modeling, for instance—would give model teams a stable, extensible interface to simulation data, creating a feedback loop from simulation to model training and back again.

On the architectural side, the article distinguishes between two main families of models suited to scientific simulation. For regular grids such as latitude-longitude climate meshes, Vision Transformer-style encoders use patch embeddings that align naturally with GPU tensor cores and batched input-output operations, facilitating large-scale spatial correlations with predictable memory layouts. For spherical or unstructured domains, graph neural networks encode locality and adjacency through message passing, preserving mesh topology without expensive remeshing. Temporal structure is typically handled by autoregressive or time-conditioned transformers, though the authors caution that long temporal windows significantly increase attention memory, communication overhead, and checkpoint sizes—all key bottlenecks on distributed supercomputing systems.

Two design principles run through the article’s architectural recommendations. First, physics-aware inductive biases—constraints that force models to respect invariants, conservation laws, and symmetries—improve data efficiency and reliability across different physical regimes. Second, token- and compute-adaptivity allows cost to scale with solution complexity rather than nominal grid size, using mechanisms such as windowed or hierarchical attention for long-range dependencies, sparsity and mixture-of-experts routing for conditional computation, and adaptive token pruning that focuses model capacity where dynamics are actually active. Linear or subquadratic attention variants further reduce the quadratic growth of standard self-attention, enabling extreme resolutions on modern accelerators.

The article also details concrete parallelization strategies drawn from ORBIT-class efforts at Oak Ridge, identifying four key levers for scaling Vision Transformer-based foundation models on supercomputers. Linear-complexity attention reduces self-attention cost from quadratic to linear, enabling multi-billion-token contexts at high resolution. Hybrid parallelization—combining tensor and model sharding with data parallelism—scales to models with over 100 billion parameters while controlling communication and memory overhead. Lightweight architectural efficiency improves hardware utilization without sacrificing accuracy, and domain specialization with robust uncertainty quantification maintains physical consistency in multi-variable predictions. The authors acknowledge that these techniques trade exact global attention for sparse routing and introduce modest auxiliary indices and buffers, and they stress that accuracy-cost impacts must be evaluated explicitly rather than assumed away.

Looking beyond transformers, the article highlights an emerging alternative: pairing diffusion-based generative models with state-space architectures such as Mamba to target linear-time complexity and lower GPU memory requirements. Mapping one-dimensional Mamba architectures to two- and three-dimensional scientific fields uses multi-directional and zigzag scanning patterns, while hybrid designs retain a small fraction of self-attention to preserve global coherence. These approaches reduce activation and key-value cache traffic, ease pressure on interconnects, and enable longer contexts on fixed high-bandwidth memory budgets—naturally improving the scalability of diffusion-based foundation models for probabilistic forecasting, ensemble generation, and uncertainty quantification in chaotic systems.

Performance on supercomputers, the authors observe, is often limited more by memory capacity, memory bandwidth, and collective communication than by raw floating-point throughput. Training costs for multi-channel two- and three-dimensional fields over long contexts are dominated by activation and key-value cache residency, optimizer state footprints, tensor layout conversions, and synchronization overhead. To address this, the article catalogs a battery of optimization techniques: fused attention kernels that reduce high-bandwidth memory movement, hybrid parallelism that shards parameters, gradients, and optimizer states across devices, key-value cache management strategies such as PagedAttention that improve batching efficiency during inference, and topology-aware placement combined with tensor fusion and overlapping collectives that convert theoretical peak performance into sustained throughput.

Cross-layer co-design emerges as a unifying theme. Sequence parallelism and tiling constrain activation footprints; activation checkpointing trades computation for memory; gradient accumulation increases effective batch size; and conditional computation through sparse activations and mixture-of-experts scales cost with solution complexity. Reliability receives particular attention: mixed and low-precision training requires careful gradient scaling and clipping for numerical stability, while deterministic kernels support fast recovery and reproducibility. HPC-native task-based runtimes such as PaRSEC orchestrate workflows as dependency graphs, overlapping collective communication with computation and prioritizing latency-critical inference alongside throughput-driven training.

The article closes with a discussion of trust, arguing that adoption ultimately depends on rigorous validation. The authors call for comparing foundation model outputs against traditional numerical solvers and observed data, quantifying uncertainty through ensembles or diffusion techniques, and enforcing physical constraints. They propose a standardized evaluation framework incorporating accuracy, efficiency, and trustworthiness metrics—mirroring the verification, physics consistency checks, and cross-solver benchmarking with confidence intervals already standard in numerical simulation—alongside a hardware-aware simulation platform for evaluating pipelines across computational architectures with pre-registered acceptance thresholds. Understanding the tradeoff between accuracy and computational time, they conclude, will let practitioners allocate resources effectively and make informed decisions about when foundation models can deliver swift, credible scientific insight on production supercomputers.

Subject of Research: Scalable foundation models for accelerating numerical simulations on high-performance computing platforms

Article Title: Scalable foundation models for numerical simulations on HPC platforms

Article References: Wang, D., Gong, Q., Liu, Z., Wang, X., Cao, Q., & Klasky, S. (2026). Scalable foundation models for numerical simulations on HPC platforms. Frontiers in High Performance Computing, 4, Article 1778471. https://doi.org/10.3389/fhpcp.2026.1778471

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2026.1778471

Keywords: foundation models, high-performance computing, numerical simulation, exascale computing, Vision Transformer, graph neural networks, diffusion models, lossy compression, AI-ready data, uncertainty quantification, parallelization, scientific machine learning