vToken is a token-level virtualization layer for LLM serving that decouples logical token liveness from physical block placement. It maintains a stable logical token view via token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. Implemented in vLLM and evaluated with H2O, Random, and Scissorhands across models, vToken reduces retained KV blocks per request by 27.2%–72.3% and improves SLA-constrained throughput by up to 1.37× compared with a paired Naive-Evict baseline. Under a constrained active-KV budget, it extends maximum feasible concurrency by up to 2×.
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. vToken, a lightweight token-level virtualization layer, decouples logical token liveness from physical block placement. It maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. Implemented in vLLM and evaluated with H2O, Random, and Scissorhands across models, vToken reduces retained KV blocks per request by 27.2%–72.3% and improves SLA-constrained throughput by up to 1.37× compared with a paired Naive-Evict baseline. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2×, while reducing the per-policy integration footprint.
vToken introduces token-level virtualization to address intra-block fragmentation in KV cache management. By using token-table indirection, it decouples logical token liveness from physical block placement, allowing finer-grained reclamation than block-level PagedAttention. Asynchronous repacking of live tokens enables physical reclamation without disrupting the logical token view. The approach preserves compatibility with existing PagedAttention kernels and CUDA Graphs, suggesting low integration overhead. Evaluation shows significant reductions in retained KV blocks and improvements in throughput and concurrency, indicating that token-level management can unlock memory efficiency gains beyond block-level allocators.
KV cache memory is a key constraint for LLM serving throughput and cost. vToken's token-level virtualization offers a practical path to improve memory utilization without requiring changes to existing attention kernels or serving frameworks. Its implementation in vLLM, a widely used serving system, suggests potential for rapid adoption. The reported improvements in SLA-constrained throughput and maximum feasible concurrency could translate into lower serving costs and better resource utilization for LLM providers. The reduced per-policy integration footprint may encourage broader adoption of KV eviction algorithms.
vToken can reduce the memory footprint of KV caches in LLM serving, enabling higher batch sizes and concurrency on the same hardware. This may lower per-token serving costs and improve SLA compliance for latency-sensitive applications. For LLM providers, the ability to serve more requests per GPU could improve gross margins and competitive positioning. The lightweight integration with vLLM reduces engineering effort and risk, making it attractive for production deployment.
Next signals to watch include: (1) whether vToken is merged into the main vLLM repository and adopted by major LLM serving providers; (2) independent benchmarks validating the reported throughput and concurrency gains across diverse models and workloads; (3) extension of token-level virtualization to other memory management layers, such as activation or weight caching; (4) integration with emerging KV eviction policies beyond H2O, Random, and Scissorhands; and (5) potential impact on serving cost models and pricing for LLM inference.