针对长文本生成，提出页感知调度引擎，提升消费级显卡性能
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
- 按KV头组分配计算任务，直接操作页表减少调度开销
- 在批量8的长序列场景下，解码吞吐提升最高达1.40倍
- 适合追求长上下文推理效率的开发者和部署工程师
自回归大模型服务正受限于键值缓存传输而非密集矩阵乘法。现有分页注意力系统虽减少碎片化，且FlashInfer等内核高度优化解码注意力，但最优单核实现未必是最佳服务调度：低活跃度长序列解码导致显卡利用率不足，混合序列长度则引发精确长度调用与粗粒度填充批处理间的矛盾。我们提出PersistentKV，一种面向分组查询注意力（GQA）的原生块表解码注意力引擎与页感知调度研究。PersistentKV按KV头组映射工作，直接在原生页表上执行，并引入紧凑的工作队列调度器，仅执行非空行-KV头-序列片段任务。在RTX 3060（FP16，页大小16，Hq=32，Hkv=8，d=128）上，校准后的屋顶线策略选择：小活跃批用FlashInfer，B1长序列步用PersistentKV序列拆分，支持的B8长序列步用PersistentKV工作队列调度。固定校准轨迹中的成本模型常数后，五个保留种子在B8双峰、均匀和齐普夫分布负载下，平均解码吞吐提升1.04至1.08倍，在B1桶状轨迹上达1.40倍。对B4边界情况及未校准的GQA比例，策略通过回退至FlashInfer避免性能下降。还报告了注意力+MLP计时代理和工作负载计数器，显示工作队列调度将每步启动扇出从16.00降至2.00。结果表明，任务分配是决定服务系统性能的关键变量。
原文摘要 · Abstract (English)
Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce fragmentation, and mature kernels like FlashInfer provide highly optimized decode attention. However, the best single-kernel implementation is not always the best serving schedule: low-active long-context decode can under-utilize GPUs, while mixed sequence lengths introduce tension between many exact-length launches and coarse padded batches. We present PersistentKV, a native block-table decode attention engine and page-aware scheduling study for grouped-query attention (GQA). PersistentKV maps work by KV-head group, executes directly over native page tables, and adds a compact workqueue schedule executing only non-empty row-KV-head-sequence-split tasks. On an RTX 3060 (FP16, page size 16, Hq=32, Hkv=8, d=128), a calibrated roofline-style policy selects FlashInfer for small active batches, PersistentKV sequence splitting for batch size 1 (B1) long-context steps, and PersistentKV workqueue scheduling for supported B8 long-context GQA steps. With cost-model constants fixed on calibration traces, five held-out seeds improve mean wall decode-token throughput by 1.04x to 1.08x on B8 bimodal, uniform, and Zipf-like workloads, and by 1.40x on a B1 bucketed trace. For the B4 boundary case and uncalibrated GQA ratios, the policy avoids regressions by routing to FlashInfer. We also report an attention-plus-MLP timing proxy and workload counters showing workqueue scheduling reduces launch fan-out from 16.00 to 2.00 launches per step on held-out bimodal B8. These results show that work assignment is a decisive serving-system variable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。