vLLM 与 PagedAttention 重塑 LLM 服务吞吐

大模型服务的瓶颈，常常不像训练那样写在 loss 上，而写在“下一秒还塞不塞得进一个请求”。机房夜班的主诉往往不是 loss 曲线，而是显存碎片、排队长度与超时告警。

自回归解码每吐出一个 token，就要把对应的 key 与 value 留下来。序列越长，KV 缓存越大；会话一多，显存账本比参数账本更难平。传统做法往往为每个请求预留一段连续空间，并按可能的最大长度留余量。长度事先不知道时，余量变成空洞：碎片与过度预留并存，GPU 算力还在，新请求却已经排队或直接 OOM。ChatGPT 之后，开放权重与自建推理同时升温，这个问题从实验室脚本变成生产主诉。

2023 年 6 月 20 日前后，以 UC Berkeley 等研究者为主的团队开源 vLLM，并介绍 PagedAttention：把 KV 缓存拆成可非连续映射的页，用类似操作系统虚拟内存的思路管理块表，降低碎片与预留浪费。同年 SOSP 论文系统描述设计；发布博客与论文不必当成同一天。作者在评测负载上相对 Hugging Face Transformers 等基线报告数量级吞吐提升——数字绑在具体模型、批大小与硬件上，不是永恒倍数。连续批处理与页级分配一起，让引擎少为未知长度预留空位。

vLLM 没有改模型公式。它改的是服务引擎如何安排显存与批处理。执行仍依赖模型质量、调度策略与网络；PagedAttention 只解决其中最刺眼的一截。吞吐数字会随负载过期。把 KV 缓存当成可分页资源、而不是每请求一条刚性连续带的思路，却会留下来——直到下一次显存账本再次写满。基础设施论文的胜利，往往是让某个曾经否决请求的理由，变成可以调参的对象。

开源引擎改变的是默认选择：自建推理时，团队会先问有没有 PagedAttention 一类的显存管理，而不是先接受“并发一高就扩卡”。吞吐数字会过期，问题重述会留下。服务系统史里，这类胜利通常很安静——没有发布会倒计时，只有更少的 OOM 与更短的排队。

PagedAttention 之后，KV 缓存管理成为服务引擎的核心课，而不是边角优化。后继系统可以换实现，却很难再假装“按最大长度连续预留”是唯一合理默认。安静的基础设施胜利，往往以这种方式改写默认。

The bottleneck in large-model serving is often not written on a loss curve. It is written on whether the next second can admit another request. Night-shift complaints are less often about loss than about fragmentation, queue length, and timeout alerts.

Autoregressive decoding keeps a key and value for every new token. Longer sequences enlarge the KV cache; more concurrent sessions make the memory ledger harder than the parameter ledger. Conventional practice often reserved a contiguous slab per request, padded toward a possible maximum length. When length is unknown in advance, padding becomes holes: fragmentation and over-reservation coexist, GPUs still have arithmetic capacity, and new work queues or OOMs. After ChatGPT, open weights and self-hosted inference rose together, and the problem moved from lab scripts to production’s main complaint.

Around 20 June 2023, a team including UC Berkeley researchers open-sourced vLLM and introduced PagedAttention: split the KV cache into non-contiguously mapped pages, manage block tables in the spirit of OS virtual memory, and cut waste from fragmentation and reservation. A SOSP paper that year described the design systematically; blog and paper need not share a calendar day. Against baselines such as Hugging Face Transformers, authors reported order-of-magnitude throughput gains on their loads—numbers bound to model, batch, and hardware, not eternal multipliers. Continuous batching and page-level allocation together let the engine reserve fewer empty slabs for unknown lengths.

vLLM did not change the model equations. It changed how a serving engine arranges memory and batching. Execution still depends on model quality, scheduling, and networking; PagedAttention only fixes one glaring slice. Throughput numbers expire with load. Treating the KV cache as pageable resource rather than one rigid contiguous band per request is the idea that remains—until the next time the memory ledger fills again. Infrastructure papers often win by turning a reason that used to reject requests into something that can be tuned.

Open engines change the default question: when self-hosting inference, teams ask first whether something like PagedAttention manages memory, rather than accepting “scale GPUs when concurrency rises.” Throughput numbers expire; the problem reframing remains. In serving-system history, such wins are usually quiet—no launch countdown, only fewer OOMs and shorter queues.

After PagedAttention, KV-cache management became a core serving-engine subject rather than a corner optimization. Successors may change implementations, but it is hard to pretend contiguous max-length reservation is the only sane default. Quiet infrastructure wins often rewrite defaults this way.