解决长文本生成中动态压缩与静态系统要求的矛盾，提升推理效率。
HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
- 设计分层缓存结构与对齐机制，统一动态选择与静态内存需求。
- 在10,000+令牌场景下实现吞吐量翻倍，保持高质量生成。
- 适合需要高效长文本生成的部署系统开发者。
长上下文大模型推理面临根本性冲突：头自适应压缩算法（如Top-$p$核采样）通过动态调整内存预算获得更高准确率，但现代推理引擎（如vLLM）依赖静态内存模式以利用CUDA Graphs和PagedAttention。本文提出HARD-KV，一种统一框架，弥合动态选择与静态系统约束之间的“静态-动态”不匹配。HARD-KV引入级联缓存层次结构，管理密集、稀疏和压缩三类层级中的令牌生命周期。关键提出日志校准机制，将异构重要性度量归一化至统一概率空间，实现跨异构头的稳定Top-$p$预算分配。为弥补效率差距，提供系统级解决方案，将碎片化的动态索引重写为适配高性能推理引擎的连续物理布局。在数学推理基准（AIME、U-Math）上的大量实验表明，HARD-KV相比静态基线实现最高2倍吞吐提升，同时在10,000+令牌场景下保持高保真生成。代码已开源：https://github.com/SuDIS-ZJU/HARDInfer。
原文摘要 · Abstract (English)
Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.g., Top-$p$ nucleus sampling) offer superior accuracy by dynamically fluctuating memory budgets, yet modern inference engines (e.g., vLLM) demand rigid, static memory patterns to leverage CUDA Graphs and PagedAttention. We resolve this ``Static-Dynamic'' mismatch with HARD-KV, a unified framework that that bridges dynamic selection with rigid system constraints. HARD-KV introduces a Cascade Cache hierarchy, managing the token lifecycle across dense, sparse, and condensed tiers. Crucially, we propose a Logits Calibration mechanism that normalizes diverse importance metrics into a unified probability space, enabling consistent Top-$p$ budgeting across heterogeneous heads. To bridge the efficiency gap, we offer a system-level solution, which rewrites fragmented, dynamic indices into contiguous physical layouts compatible with high-performance inference engine. Extensive experiments on math-reasoning benchmarks (AIME, U-Math) verify that HARD-KV achieves up to 2$\times$ throughput improvement over static baselines while maintaining high-fidelity generation in 10k+ token scenarios. Code is available at https://github.com/SuDIS-ZJU/HARDInfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。