Agent Workloads Expose the Limits of Round-Robin
The Google team has released llm-d 1.0, an open-source, Kubernetes-native distributed inference stack that coordinates vLLM or SGLang model servers as a shared serving pool. It targets self-hosted agents and reinforcement-learning workloads, addressing a concrete weakness of ordinary round-robin load balancing: when requests share context and traffic becomes uneven, a simple balancer has little basis for choosing where the next request should go.
An agent rarely sends a single prompt and stops. It makes a sequence of requests, and context from an earlier turn may still reside in one replica's KV cache; sending the next turn elsewhere can mean processing that context again. Bursty agent loops can also consume queue and accelerator capacity, delaying interactive requests. The technically important move in llm-d 1.0 is to treat both context reuse and fairness between tenants as scheduling problems for the serving layer.
The Router Starts Looking at Context
Built on the Kubernetes Gateway API Inference Extension, the llm-d router tracks which replicas hold particular cached prefixes, then considers load and predicted latency when choosing a destination. From the caller's perspective, an agent still uses one service endpoint. Inside the system, each turn has a chance to reach a replica that already holds relevant context. Unlike a balancer that counts only current requests, this scheduler also asks whether previous work can be reused.
Context does not have to remain in accelerator memory indefinitely. The described cache path can move KV cache from full accelerator memory to host memory, then to SSD or Managed Lustre. The integrations named in the material include vLLM's OffloadingConnector, Mooncake, and SGLang's HiCache. The deployment envelope is also explicit: llm-d 1.0 supports vLLM and SGLang on NVIDIA, AMD, or Intel GPUs and Google TPUs, with cache tiers spanning HBM, CPU memory, SSD, and Lustre. Whether those tiers help in practice still depends on the team's serving stack, hardware, and storage configuration.
Scaling the Pool Also Means Splitting the Work
llm-d also runs prefill and decode in separate resource pools. Prefill is primarily compute-intensive, while decode is more sensitive to memory bandwidth and state access. Separating them lets teams scale each stage independently instead of allowing pressure on one to directly constrain the other. llm-d is not a new model execution engine; it adds cluster-level routing, cache, and deployment coordination on top of vLLM and SGLang.
That makes the focus of version 1.0 less about adding hardware support and more about bringing distributed serving closer to production operations. The listed capabilities include v1 APIs, a hardened request path, router hot standby and hot updates, and coordinated rolling upgrades of prefill and decode pools through DisaggregatedSet, designed to avoid losing sessions across mismatched versions. Red Hat, IBM, and Google are the lead maintainers; the project is built around Kubernetes and released under Apache 2.0. These choices position it as a composable infrastructure layer rather than a plugin tied to one model server.
Fair Scheduling and Idle Capacity Target Different Waste
Multi-tenant controls address a quality-of-service conflict: bursts from agent workloads should not crowd out latency-sensitive interactive traffic. llm-d 1.0 adds priority- and fairness-aware flow control, along with session-aware routing, turn-priority scheduling, and native Responses API state support. The point is not that every request becomes faster. It is that the system gains ways to distinguish request priorities and preserve session context while scheduling work.
A separate path sends asynchronous jobs to capacity that would otherwise be idle. The llm-d-async-processor can pull jobs from Google Cloud Pub/Sub or Redis and run inference when capacity is available; the material also describes inserting inference into idle reinforcement-learning cycles. Google reports that this time-slicing raises accelerator duty cycle from about 40% to above 70%. That result also marks a limit: utilization above 70% still leaves roughly 30% idle, and filling that slack depends on whether jobs can tolerate delay, whether queues have work, and whether the resources are actually free.
Read the Performance Claims in Their Test Context
The performance figures in the material are self-reported by Google and llm-d adopters. They are useful as hypotheses to test, not as capacity-planning parameters that can be applied directly. For an Ironwood TPU (TPU 7x) coding workload, Google reports that prefix-aware routing plus host-memory offload reduced median TTFT by more than 85% and delivered 2.4 times the throughput of a plain Kubernetes Service. On the Gemini platform, the reported prefix-cache hit rate rose from 35% to 70%, while coding-agent TTFT fell by more than 35%; under bursty chat traffic, P95 TTFT fell by more than 50%.
These results cannot be reduced to a promise that installing llm-d will make any workload faster. Cache hits, offloading, and prefill/decode separation all depend on workload characteristics and system configuration, while the chosen baseline also matters. The material mentions a published benchmark on a 17,000-node, multi-region GKE deployment, but the supplied excerpt does not provide enough methodology to reproduce that test. A safer decision for technical leaders is to first establish whether their traffic has reusable prefixes, bursty multi-tenant demand, or idle periods, then compare end-to-end latency, throughput, and utilization on the same hardware and service objectives. If those conditions are absent, another distributed scheduling layer may add operational surface area without reducing cost.