Helios cuts MoE serving latency with congestion-aware expert placement

Mixture-of-Experts language models can reach trillion-parameter scale while activating only a sparse subset of parameters for each token, but serving them across distributed GPU clusters creates severe communication bottlenecks during expert routing. A new study in Cluster Computing introduces Helios, a two-phase framework designed to ease those bottlenecks by combining an offline optimization step with continuous runtime adaptation.

The first phase computes a robust initial expert layout using a congestion-aware Integer Linear Program that directly optimizes for link bandwidth and transmission latency. According to the authors, existing expert placement strategies rely primarily on static offline optimizations based on simplistic hop counts, which makes them ineffective against dynamic workload shifts and real-world network congestion. By encoding bandwidth and latency into the placement problem itself, Helios aims to produce layouts that hold up better once traffic patterns change.

The second phase is a lightweight runtime adaptation engine that continuously monitors expert utilization and network telemetry. When certain experts become hotspots, the engine dynamically replicates them onto underutilized GPUs, spreading the load without waiting for a full redeployment. The framework was developed by Seyed Hossein Ahmadpanah and Meghdad Mirabi, whose article appears in Volume 29 of the journal as article number 906.

The performance claims come from discrete-event simulations rather than production deployments. On Fat-Tree and Dragonfly network topologies, Helios reportedly outperformed state-of-the-art static baselines, delivering up to 45 percent higher goodput and a 52.6 percent reduction in p99 latency under non-stationary workloads. Goodput measures useful delivered throughput, while p99 latency captures the response time that 99 percent of requests fall under, so the reported gains target both average efficiency and worst-case tail behavior.

The emphasis on non-stationary workloads is central to the argument. Static placement methods assume that traffic to experts stays roughly stable, but real MoE serving can shift as inputs change, concentrating demand on a few experts at a time. Helios addresses this mismatch in two ways: the ILP-based initial layout anticipates congestion at the network level, and the replication engine responds to observed shifts during operation.

The simulation scope matters for interpretation. Fat-Tree and Dragonfly are standard data center topologies, and discrete-event simulation allows controlled comparison across configurations, but the results do not yet demonstrate behavior on physical clusters with real hardware faults, competing jobs, or production traffic. The reported percentages describe simulated performance against static baselines under the tested conditions, not guarantees for every deployment.

Even so, the work speaks to a growing constraint in large-model infrastructure. As MoE architectures scale, the cost of moving tokens to the right experts across a network can dominate serving time, making placement and routing decisions as consequential as the models themselves. A framework that treats congestion as a first-class optimization target, and replication as a runtime lever, offers one concrete response to that constraint.

For operators of GPU clusters, the practical takeaway is that expert placement need not be a one-time offline decision. A layout computed with bandwidth and latency in mind, paired with telemetry-driven replication of hotspot experts, may keep serving performance closer to its peak as workloads drift. Whether the simulated gains translate to production systems will depend on follow-up work beyond the scope of this study.

Subject of Research: A congestion-aware placement and dynamic expert replication framework for scalable mixture-of-experts model serving in distributed GPU clusters.

Article Title: Congestion-aware routing and dynamic replication for scalable mixture-of-experts inference

Article References: Ahmadpanah, S. H., & Mirabi, M. (2026). Congestion-aware routing and dynamic replication for scalable mixture-of-experts inference. Cluster Computing, 29(16), Article 906. https://doi.org/10.1007/s10586-026-06732-3

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06732-3

Keywords: mixture-of-experts, expert routing, GPU clusters, congestion-aware routing, dynamic replication, p99 latency, goodput, Fat-Tree topology, Dragonfly topology, integer linear programming, distributed inference, Congestion-aware