Serverless Cloud Functions Close In on Supercomputer Performance for Machine Learning Data Work

For more than a decade, the heavy lifting behind artificial intelligence has been done on clusters of machines owned, cooled, and cabled by the organizations that use them. Data engineering, machine learning training, and large-scale preprocessing have traditionally demanded dedicated high-performance computing infrastructure: racks of servers connected by low-latency interconnects, tended by specialized staff, and paid for whether or not they are actually running. A new study from researchers at the University of Virginia and its Biocomplexity Institute, published in Frontiers in High Performance Computing, argues that a very different kind of infrastructure, the humble serverless function, can now do much of this work at nearly the same speed, provided one long-standing bottleneck is engineered away.

Serverless computing, exemplified by Amazon Web Services’ Lambda platform, is the cloud’s most hands-off offering. Users upload code that is chopped into individual functions, and the provider handles provisioning, scaling, and maintenance behind the scenes. Billing is metered in milliseconds, and horizontal scaling is effectively unlimited. For embarrassingly parallel tasks, such as crunching thousands of independent image tiles or sequencing records, Lambda has already proven attractive. The catch has always been communication. When many functions must cooperate on a single tightly coupled computation, such as a distributed join across terabytes of tabular data, they cannot talk directly to one another. Instead, developers have funneled intermediate data through external stores like Amazon S3 or DynamoDB, which are dramatically slower than the direct, memory-to-memory messaging that defines high-performance computing.

The Virginia team, led by Mills Staylor and Arup Kumar Sarker with co-authors including Geoffrey Fox and Judy Fox, set out to eliminate that detour. Their foundation is Cylon, a high-performance distributed dataframe library written in C++ with Python bindings, which grew out of earlier work on the Twister2 toolkit. Cylon follows the Bulk Synchronous Parallel model, the same architectural pattern that underlies classic MPI programs: every worker executes the same program on its own slice of data, and all workers synchronize at regular barriers between supersteps. On a traditional supercomputer, the Message Passing Interface handles that synchronization effortlessly. On serverless platforms, where each function hides behind a network address translator and AWS forbids the system calls needed to configure conventional transports, it simply did not work.

The researchers’ solution borrows a trick from networking known as NAT traversal, or TCP hole punching. Because each Lambda function sits behind a NAT gateway that rewrites internal addresses into a shared external address, two functions cannot directly address each other. Hole punching coordinates the pair through a publicly reachable rendezvous server: each function opens a connection outward to the server, which relays the external address and port of each endpoint to the other. When both sides then attempt simultaneous outbound connections, the NAT gateways, already primed with translation table entries, allow the packets through, and a direct TCP channel opens between the two functions. The team built this capability into a custom communicator for Cylon, inspired by and extending the FMI library, a fast message-passing system for serverless functions previously described by Copik and colleagues.

The engineering was far from trivial. An earlier attempt to run Cylon on AWS Fargate by modifying the UCX communication library’s transport layer succeeded briefly before AWS architecture changes forced a deprecation, and Lambda’s restrictions made a UCX-based approach impossible. The team instead recreated and validated FMI’s functionality as a custom Python Lambda container image, then added features the original library lacked: an AllToAll collective needed for distributed joins, variable-length gather operations, non-blocking I/O, socket retry logic, ping messages to prevent premature socket teardown, and rank assignment via atomic counters, with Redis locks imposed to prevent race conditions during blocking operations. Orchestration is handled by AWS Step Functions, which spawn a configurable number of executor functions, each pulling its workload script from S3 and connecting through the rendezvous server before the computation begins.

To benchmark the design, the team ran weak and strong scaling experiments on a distributed dataframe join across three platforms: AWS Lambda, AWS EC2 virtual machines, and Rivanna, the University of Virginia’s HPC cluster. Weak scaling used 9.1 million rows per node, while strong scaling used 4.5 million, a limit imposed by Lambda’s maximum of 10,240 megabytes of memory per function. Every platform ran the same Dockerized runtime, with the HPC experiments executed inside Singularity containers via Apptainer, ensuring a consistent software environment across wildly different hardware. On Rivanna, the team even swapped MPI for the more portable UCX and UCC libraries, pairing UCX for point-to-point messaging with UCC for collectives, an integration the authors say has not previously been applied to machine learning data pipelines.

The headline result is striking. In strong scaling, Lambda achieved a 15.85-fold speedup at 64 nodes relative to its own single-node baseline, compared with 16.96-fold for EC2, meaning serverless scaling efficiency landed within 6.5 percent of provisioned virtual machines. At intermediate sizes, Lambda sometimes even outpaced EC2, posting a 23.3 percent advantage at 8 nodes. Rivanna was faster in absolute terms, largely because its newer Intel Xeon Gold processors deliver roughly 40 percent better instructions per cycle than the older Ivy Bridge chips in the m3 EC2 instances the team tested, but the scaling ratios, which are the fair basis for comparison, were essentially equivalent across paradigms. The message is that tightly coupled parallel computation, long considered out of reach for serverless, is now feasible there.

The communication benchmarks explain why. Direct TCP through NAT traversal achieved 10 to 100 times lower latency than storage-mediated alternatives. At 32 nodes, the direct approach completed the join in roughly 60 seconds, against 255 seconds for Redis-mediated messaging and 455 seconds for S3, whose per-object upload and download round trips punish every data shuffle. Microbenchmarks of MPI-style collectives showed AllReduce completing in about 13 milliseconds at 32 nodes, with barrier synchronization scaling logarithmically from 0.9 milliseconds at two nodes to 7 milliseconds at 32, fast enough for BSP workloads that synchronize frequently. A GroupBy aggregation operator with a combiner optimization scaled from 20.1 seconds on one node to just 27.1 seconds on 32 nodes even with 50 million rows per node, showing the architecture generalizes beyond joins.

Cost is the other half of the story, and it is where serverless shines for bursty, intermittent workloads. Because Lambda bills per gigabyte-second of actual execution, a 32-worker distributed join cost approximately $0.032 on Lambda, compared with an estimated $0.238 for the equivalent EC2 fleet billed by the hour, a roughly 7.4-fold saving when idle time dominates. The entire suite of 120 experimental executions cost just $3.25. Orchestration overhead from Step Functions proved negligible. For organizations whose data pipelines run sporadically, paying for a perpetually warm cluster can be hard to justify; paying only while functions execute changes the calculus entirely.

Limitations remain, and the authors are candid about them. Cylon currently implements only about 30 percent of Pandas’ operator set, Lambda functions time out after fifteen minutes, and the tightly synchronized BSP model means a single failed worker kills the whole job, with no checkpointing yet available, though Step Functions retries and resiliency mechanisms in the hole-punching protocol, including exponential backoff with jitter and connection validation handshakes, harden the fragile setup phase. The team plans Cython-based convergence with the full Pandas API, Twister2-style checkpointing for fault tolerance, libfabric support for GPU-backed cloud communication, and real-world deployments in the domains that motivated the work: genomics projects measured in exabytes, hydrology and earthquake time-series forecasting, and serverless astronomical image processing. If those plans bear out, the assumption that serious machine learning data engineering requires owned hardware, or even rented virtual machines, may soon look like a relic of the field’s early days.

Subject of Research: Integrating serverless computing with high-performance distributed dataframes for machine learning data-intensive workloads

Article Title: Combining serverless and high-performance computing paradigms to support ML data-intensive applications

Article References: Staylor, M., Kumar Sarker, A., von Laszewski, G., Fox, G. C., Cheng, Y., & Fox, J. (2026). Combining serverless and high-performance computing paradigms to support ML data-intensive applications. Frontiers in High Performance Computing, 4, Article 1767201. https://doi.org/10.3389/fhpcp.2026.1767201

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2026.1767201

Keywords: serverless computing, AWS Lambda, high-performance computing, distributed dataframes, machine learning, NAT traversal, TCP hole punching, Cylon, bulk synchronous parallel, cloud computing, data engineering, FaaS