llm-d was born Kubernetes-native. Its workers are Deployments, its endpoints live in an InferencePool, and its guides assume a cluster is one kubectl away. That made sense: Kubernetes is where most production inference runs, and building on it gave llm-d a head start on networking, lifecycle, and scale.

But the thing that makes llm-d llm-d - KV-cache-aware scoring, prefix-cache affinity, prefill/decode disaggregation, flow control - was never fundamentally about Kubernetes. It is routing intelligence. It reasons about the state of a fleet of model servers and decides where each request should go. Nothing about that logic needs an API server. The dependency on Kubernetes was incidental, inherited from how endpoints happened to be discovered, not essential to what the router actually does.

This post is about pulling those two things apart. We introduce the EndpointDiscovery abstraction in the llm-d router that separates what endpoints exist from how to route across them, and the first plugin built on it - file discovery - which lets the full routing stack run as a plain process or container with no Kubernetes anywhere in sight: on an HPC cluster, inside a Ray job, on a bare-metal rack, or on your laptop.

Figure 1: The big picture - one routing stack under every platform. llm-d discovers endpoints through its EndpointDiscovery module (Kube Discovery against an InferencePool, File Discovery against everything else) and serves requests the same way on Kubernetes, Slurm, Ray, or bare metal (inference, HPC, and RL rollout workloads: veRL, SkyRL, prime-rl). The rest of this post explains how.

The math behind Reinforcement Learning (RL) post-training for Large Language Models is notoriously unforgiving. As frontier AI labs push the boundaries of reasoning and coding models using algorithms like GRPO, they routinely hit hard architectural and physical constraints. While much of the industry's focus remains on raw GPU count, the actual gatekeeper of experimental velocity is infrastructure efficiency.

As established in large-scale systems research including ByteDance’s HybridFlow architecture (powering veRL) and Alibaba’s RollMux, optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of RL Total Cost of Ownership (TCO). While these frameworks define the architecture, executing this at production scale is challenging. We foresee this trend early on and have been purposefully building llm-d to address efficiency bottlenecks in RL. llm-d is not a reactive patch, it is a mature, production tested engine that is being deployed across production RL workloads.

We have made targeted investments to expand llm-d into a comprehensive, composable RL infrastructure stack designed to meet researchers where they are i.e. whether they are using Slurm or Kubernetes based setups:

1. Throughput-Driven Inference: The llm-d router maximises the RL rollout generation throughput, ensuring samplers can saturate the pipeline and prevent downstream optimization trainers from stalling.
2. Agent Sandbox (Sister Project): Fast tool use and isolated code execution are vital for generating reward signals. Our Agent Sandbox provides the secure, sub-second execution environments necessary to scale verifiable rewards and agentic rollouts without bottlenecking the GPUs.
3. Core Pipeline Primitives: To combat generation stragglers, we shipped asynchronous batching and native inference schedulers, specifically designed to orchestrate complex, multi-turn RL workloads.

Today, we are introducing a new well-lit path in the llm-d project for Co-operative Time-Slicing : Snapshot Agent. Time-Slicing effort enables job interleaving into the Kubernetes platform layer to address the single greatest source of waste in modern RL: the Idle Accelerators. Snapshot agent is the first component that we are launching, other components will soon follow.

The following two sections in the blog provide a high level overview of problems in running RL workloads on scale followed by a brief overview of investments for solving those problems and then we dive deeper into the time-slicing for RL.

Core Challenges in Enterprise RL Infrastructure​

Deploying large-scale RL infrastructure presents operational friction and cost inefficiencies. Main inefficiency among these is low hardware utilization; because typical RL training loops suffer from alternating blocking phases leading to an average GPU/TPU duty cycle of only around 40%. This structural waste is exacerbated during the sampling step, where generation stragglers create significant tail latencies and drag down overall sampling efficiency.

Furthermore, scaling distributed RL remains an engineering bottleneck. Teams face difficulties in configuring network primitives like NCCL and Intra-Cluster Interconnect (ICI), while simultaneously battling slow checkpointing operations and high-latency weight transfers across decoupled pools.

Finally, as organizations pivot toward Agentic RL use cases, they need isolation for untrusted code execution and tool calling during generation and reward evaluation steps during RL, all while struggling with reliability when running RL in distributed environments at scale.

The Solution : A composable RL Infrastructure Stack​

To resolve these bottlenecks, llm-d have been investing in improving the efficiency, security, and observability for RL workloads. The solutions are composable meaning you can adopt one or more depending on your use case.

Efficiency:

• Co-operative Time-Slicing (repo, well-lit path) : We are introducing platform-level GPU and TPU time-slicing to multiplex concurrent RL jobs onto shared physical hardware, driving accelerator duty cycles from the poor 40% baseline up to a saturated 95%+ efficiency.
• RL scheduler (repo) : To eliminate tail latency, the platform implements intelligent routing and batching mechanics for the sampling step, boosting overall throughput up to 20% samples per second per accelerator.
• Weight Propagation interface (repo) : Weight transfer between sampling and training is streamlined via built-in Kubernetes controllers that automate NCCL,NIXL and ICI topologies, providing a faster and kubernetes native weight-transfer interface.

Security & Observability

• Sandboxing: To mitigate the risks of Agentic flows with RL, tool calling and dynamic code execution are isolated with Agent Sandboxes.
• Telemetry: Wrap these capabilities with a managed RL monitoring dashboard—delivering out-of-the-box golden telemetry, metrics, and distributed traces—and enterprises gain instant, actionable visibility to seamlessly troubleshoot and scale their most demanding RL pipelines.

The Core Efficiency Problem with RL Loops​

Distributed RL post-training, is a highly fragmented "stop-and-wait" loop. The pipeline operates as a continuous cycle between Generation and Optimization. While generation is running optimization accelerators are idle and vice-versa.

This structural cadence introduces two massive systemic inefficiencies at scale:

• The Idle Accelerators: Because these phases occur sequentially, expensive GPU and TPU clusters sit completely idle (0% utilization) for 40% to 60% of their lifecycle. Trainers sit idle waiting for sampling rollouts to finish; samplers sit idle during gradient updates and weights distribution. This "deadtime" could represent millions in wasted capital annually.
• The Context is Locked-In: RL training and samplers hold their accelerator allocations for the entirety of their runtime even during idle phases because the CUDA context and all device memory remains resident. Standard schedulers treat these pods as static, siloed allocations, completely rather than aligning to the alternating, phase-level states of the live RL loop.

Introducing Co-operative Time-Slicing (RL Job interleaving)​

To eliminate idle accelerators during RL jobs, we are introducing Co-operative Time-Slicing under the llm-d project. Rather than forcing hardware to wait on upstream phases, the infrastructure dynamically interleaves independent RL jobs onto shared hardware blocks, driving aggregate accelerator duty cycles up to 95% without altering the underlying model convergence or accuracy.

When Job A pauses its training phase to run rewards evaluation on the CPU or distribute updated weights, the infrastructure time-slices the physical accelerators, swapping in the active sampling or training phase of Job B.

High Level Architecture Overview​

The Accelerator Time-Slicing Platform architecture is divided into three distinct operational boundaries—Workload-scoped, Cluster-scoped, and Node-scoped—to isolate developer code, cluster coordination, and physical hardware management. This layout maps how user-space runtime requests are translated into cluster-level lock queues and executed as node-level process context swaps.

1. Workload-Scoped Layer (Application Runtime)​

This top layer encapsulates the containerized user training environment, such as a dedicated RayCluster or standalone Kubernetes pod configuration. It isolates the modeling code from infrastructure complexity:

• RL Loop Actor Process: The central coordinator of the training loop. RL code imports Timeslice Client library, which explicitly signals phase boundaries to the control plane using simple acquire() and yield() functions.
• ML Framework Worker Process: The background compute worker running the core training stack (e.g., PyTorch, vLLM). It maintains the live CUDA context, model weights, and allocations inside its virtual memory address space.

2. Cluster-Scoped Layer (Control & Orchestration Plane)​

This middle layer governs cluster wide state, scheduling policies, and queue coordination across multi-tenant workloads:

• Workload Scheduler: A native Kubernetes scheduling infrastructure configured with Dynamic Resource Allocation (DRA). It uses custom DeviceClass parameters to enforce device oversubscription, instructing the cluster that it is safe to co-schedule multiple trainer/sampler pods onto the same physical hardware footprint.
• Accelerator Orchestrator: The central brain of the platform. It dynamically discovers group topology based on user-defined labels. It maintains FIFO lock queues per group to coordinate access to the accelerator and fans out atomic snapshot or restore commands to the Snapshot Agent on the target nodes.
• Workload Placement Optimizer: A future component that runs alongside the orchestrator. It automatically profiles workload execution patterns and configures time-sliceable job groups dynamically without manual user labeling.

3. Node-Scoped Layer (Hardware & Data Plane Isolation)​

This bottom layer acts on the physical node boundary to enforce absolute memory isolation between oversubscribed pods:

• Snapshot Agent DaemonSet: A privileged daemon running on every accelerator node, exposing a gRPC interface that handles snapshot and restore calls from the Orchestrator or directly from an RL service. It performs the accelerator state save and restore.
• Pluggable Save Backend: A modular interface that translates agent commands into host process manipulation. The v1 implementation uses a cuda-checkpoint binary and NVML cgroup tracking to discover compute processes, freeze execution, and serialize the entire active CUDA context directly to host DRAM.
• Physical Accelerator Hardware Pool: The underlying physical GPU/TPU cluster infrastructure where oversubscribed execution steps alternate seamlessly without OOM risks or framework-level interference.

End-to-End Control and Data Flow for Time-Slicing​

1. The user's RL training loop reaches a natural pause point—such as the end of a rollout generation phase or a training step, a logical boundary where it is safe to give up the physical accelerator. At this boundary, the job invokes the TimeSlice client library to request hardware access for its next operational phase.
2. The client library dispatches an Acquire RPC to the Accelerator Orchestrator, explicitly identifying the active job_id and the targeted group of hardware nodes it requires. This remote procedure call synchronously blocks, forcing the workload thread to wait until the platform layer determines it is safe to proceed.
3. If another independent RL job currently holds the execution lock on the designated accelerator pool, the Accelerator Orchestrator places the incoming request into a First-In, First-Out (FIFO) queue assigned to that node group, preserving the strict order of request arrival.
4. When the active job finishes and yields control, the orchestrator begins a coordinated swap. It communicates with the node-local Snapshot Agent to trigger a memory snapshot of the yielding job's active accelerator state, serializing its entire live CUDA context and moving the data off the physical accelerator into host DRAM.
5. With the physical accelerator hardware now successfully evacuated and free of memory residency, the orchestrator evaluates the queue state, pops the next pending job from the front of the FIFO line, and initiates the control transfer.
6. The orchestrator coordinates with the destination node's Snapshot Agent a second time, executing a restore operation. The agent deserializes the incoming job's cached state, lifting its model weights and execution context out of host DRAM and streaming them back onto the accelerator's high-bandwidth memory (VRAM).
7. Once the physical memory restore completes successfully, the orchestrator grants the execution lock to the new job and returns a success status to the blocked Acquire call. The worker process immediately unblocks and resumes execution exactly where it left off, requiring zero rewrites or modifications to its core modeling logic.
8. The job that previously yielded its compute window now sits warm in host memory. Its process state remains active, completely bypassing container cold-start overhead and standing ready to be re-streamed onto the accelerators just as efficiently when its turn comes back around in the scheduling queue.

Early Benchmarks​

To validate the infrastructure’s ability to reclaim stranded compute capacity without impacting algorithmic convergence, early tests evaluated the platform-native time-slicing system using a representative Reinforcement Learning (RL) workload featuring veRL/GRPO, PyTorch FSDP trainers, and vLLM samplers on NVIDIA H100 GPUs

During the test, interleaving two independent sampler workloads on a single node elevated the actual hardware duty cycle from a baseline of 41% to 71%, with a theoretical peak of ~95% under idealized phase alignments. Because the active job has exclusive access to the GPU during its compute window, there is zero degradation to token generation or training step throughput.

Simple Developer Experience (client-side)​

We believe researchers should focus on core modeling logic rather than wrestling with low-level CUDA context switching or custom scheduling loops. With llm-d, you can enable RL job multiplexing by wrapping the accelerator phases in your existing loops with a simple Python decorator:

from timeslice import OrchestratorClient

orchestrator = OrchestratorClient(job_id="job-1")

@orchestrator.on_accelerators(group_id="trainer-group")
def train_phase(model, trajectories):
    return model.update(trajectories)

@orchestrator.on_accelerators(group_id="sampler-group")
def generate_phase(model, prompts):
    return model.generate(prompts)

# Standard sequential loop — interleaved with other jobs under the hood
for epoch in range(EPOCHS):
    trajectories = generate_phase(policy, dataset)
    rewards = compute_rewards(trajectories)
    train_phase(policy, rewards)

Current Release and Future Outlook​

Today we are releasing the Snapshot Agent along with a well-lit path for integrating it into managed multi-tenant RL services to support full fine-tuning.

Here's what comes next.

Expanding the Snapshot Agent: We will add backends that enable ultra-fast snapshots and selective snapshot support that checkpoints only specific memory regions such as LoRA adapter weights. Together, these unlock faster context switches for full fine-tuning workloads and open up new use cases such as multi-tenant RL services that time-slice between tenants by swapping only the adapter state.

Accelerator Orchestrator and Timeslice Client: The Orchestrator coordinates time-slicing across multiple jobs managing cooperative lock queues and driving context switches at phase boundaries. Alongside it, we will ship a lightweight client library that any RL application can integrate with using a two-call API: acquire() before the accelerator phase and yield() after.

Framework integrations: We will provide well-lit paths for integrating with slime, veRL, and other popular RL training frameworks making it easy for researchers to adopt time-slicing without leaving their existing training stack.

Simplified onboarding and intelligent job placement: We will simplify platform deployment and, looking further ahead, build a component that automatically identifies time-sliceable workloads and makes intelligent job placement decisions eliminating the need for users to manually identify and group compatible workloads.

Expanding accelerator support: In line with llm-d's hardware-agnostic mission, we are designing the platform to span multiple accelerator architectures starting with GPU and expanding to TPU and beyond.

Help Shape the Future of SIG-RL​

Building robust, highly optimized RL infrastructure requires tight collaboration with the engineers and researchers running these workloads at scale.

If you are currently wrestling with low GPU utilization, synchronization stalls, or complex scheduling logic in your post-training pipelines, we want your feedback:

• Use Time-Slicing for your RL jobs - [repo](https://github.com/llm-d-incubation/llm-d-rl-time-slicing/tree/main.
• Explore the Code: Visit our repositories under the llm-d GitHub organization.
• Join the Discussion: Join the #sig-rl channel in the llm-d Slack.
• Contribute: Share your reference implementations, benchmarks, and edge cases to help us refine this well-lit path.

Networking: The Critical Path in P/D Disaggregation​

llm-d's prefill-decode disaggregation unlocks significant efficiency gains by separating compute-heavy prefill from memory-bandwidth-heavy decode onto dedicated GPU pools. But it introduces a hard dependency on the network: the KV Cache must be transferred from prefill to decode before the first token can be generated. This transfer time lands directly on the Time to First Token (TTFT) — making networking a first-order concern for end-to-end inference latency.

This post dives into llm-d's networking stack — how it works today and how it's evolving in collaboration with NVIDIA.

For most of the transformer era, the KV cache rested on a quiet assumption: one model, one uniform cache. Every layer attended the same way, every block was the same size, and everything built on top of the cache (allocators, offload connectors, schedulers) could treat it as a single pool.

Hybrid models broke this assumption. Many recent frontier and open-weight models increasingly mix attention types within a single model (full attention next to sliding-window, linear, or Mamba layers), making the cache heterogeneous: different layers now hold different amounts of state, in different shapes, with different reuse rules. A cache block that used to be allocated as one uniform unit is now constituted of several distinct parts.

To serve a hybrid model efficiently, an AI inference platform has to handle that heterogeneity in at least three aspects of the stack:

• GPU Memory Allocation: How the cache is laid out and allocated on the GPU. vLLM solved this with its Hybrid Memory Allocator (HMA), rebuilt around a unified allocator (see Hybrid Models as First-Class Citizens in vLLM).
• KV Offloading: Extending the KV cache to CPU and storage. Without HMA awareness, an offloading connector turns the HMA off and therefore discards the GPU memory improvements or potential data movement savings.
• KV-Aware Routing: Sending each request to the right model-server replica. Ignoring hybrid memory structure may erroneously list nodes as having or not having the required KV data based on information stemming from just part of the layers.

vLLM's HMA solved hybrid GPU memory allocation when handling a single vLLM instance. This post shows how llm-d extends that to tiered KV cache management - including KV offloading to CPU and storage, and KV-aware request routing - significantly improving throughput and latency at scale for hybrid models.

Most production inference clusters today are single-vendor because that is often the simplest way to configure and operate a cluster.

That is starting to change. Procurement cycles bring new generations alongside older ones, supply planning spans multiple accelerator options, and cost/performance profiles differ by workload. Real production fleets are accumulating heterogeneity whether or not the architecture planned for it.

This is an opportunity to unlock real value: different accelerator classes can be matched to workload requirements, stranded capacity gets reclaimed, and operators gain more flexibility in capacity planning. The case is stronger still for sovereign and on-premise deployments, where data residency, regulatory alignment, and the long-term economics of high-volume inference make local fleet optimization especially important.

Making that work in practice is a non-trivial systems problem. Each accelerator stack brings its own optimized drivers, firmware, container images, runtime settings, and attention kernels. A coherent serving layer needs to preserve those platform-specific optimizations while still giving operators one control plane for routing, observability, and policy.

Deploying llm-d is not just a question of choosing a model server and adding GPUs. In a production inference deployment, operators have to choose routing policies, admission behavior, batching settings, KV-cache reuse strategies, prefill/decode placement, and autoscaling rules under concrete TTFT, ITL, throughput, and cost constraints.

These choices are coupled. A routing change that improves cache locality can concentrate load. A prefill/decode threshold that helps one workload can hurt another. An admission policy that protects critical traffic can reduce total served volume. A change in any one policy can shift TTFT, inter-token latency, throughput, SLO compliance, and accelerator cost in ways that are difficult to predict analytically.

The only reliable way to confirm those tradeoffs is to measure them in a GPU-backed llm-d cluster. But using cluster runs as the first step in every policy or capacity-planning experiment is too slow and expensive. BLIS provides a faster inner loop: a calibrated discrete-event simulator for distributed inference systems like llm-d. Developers can evaluate candidate policies and deployment configurations locally, then reserve cluster validation for the candidates most likely to matter.

If v0.6 was about proving what llm-d could do—OTel integration, prefill/decode disaggregation, initial multi-accelerator images—then v0.7 is about making sure you can actually deploy it. The theme across every category is the same: remove friction, broaden hardware reach, and give operators the documentation and CI coverage to trust the system in production.

Recent external validations demonstrated llm-d's performance gains. Those capabilities remain and continue to improve, but v0.7's investment is making them accessible: onboarding guides, tested installation paths, and confidence the guides work on your target platform.

The Problem with "Simple" LLM Deployments​

Everyone is racing to run Large Language Models (LLMs), in the cloud, on-prem, and even on edge devices. The real challenge, however, isn't the first deployment; it's scaling, managing, and maintaining hundreds of LLMs efficiently. We initially approached this challenge with a straightforward vLLM deployment wrapped in a Kubernetes StatefulSet.

Not all LLM requests cost the same. A short prompt might complete in milliseconds, while a long one can occupy a GPU for seconds. If we can predict how long a request will take on each candidate server before dispatching it, we can make substantially better routing decisions. This post describes a system that does exactly that: a lightweight ML model trained online from live traffic that replaces manually tuned heuristic weights with direct latency predictions.

llm-d is a distributed inference platform spanning multiple vLLM instances. KV cache hits are critical to achieving high inference throughput. Yet, in a distributed environment, cache hits do not occur across different nodes as the KV cache is local to each vLLM instance. In addition, this local cache is limited in size, further limiting KV data reuse. This blog presents a new way to offload KV cache to storage, tackling both aforementioned challenges – KV cache sharing and KV cache scale. llm-d's filesystem (FS) backend is a KV cache storage connector for vLLM that offloads KV blocks to shared storage based on vLLM's native Offloading Connector. While the llm-d FS backend can speed up serving of single requests (improve TTFT), its main goal is rather to preserve stable throughput and low latency at scale, as concurrency and context lengths grow. This is accomplished by significantly enlarging the cache space and enabling KV reuse across multiple replicas and nodes in llm-d.

While there are a number of existing solutions for KV cache offload to storage (e.g. LMCache or Dynamo KVBM), the new connector offers simplicity, can run with llm-d and vLLM as the only dependency, and exhibits improved performance over state-of-the-art shared storage connectors.