arXiv:2605.20868cs.LGcs.AI2026-05被引 3

让量化注意力计算可实时验证,出错时自动回退到精确结果。

Runtime-Certified Bounded-Error Quantized Attention

  • 分层缓存架构:显存存量化键值,内存存原始精度用于回退
  • 每步每头的误差有界,保证计算结果在可控范围内
  • 适合对精度要求严苛的长文本任务,如金融、医疗推理

KV缓存量化可降低大模型长上下文推理的内存开销,但引入近似误差,且现有系统仅依赖平均鲁棒性,缺乏运行时故障检测与恢复机制。本文提出分层KV缓存架构,将INT8键和INT4值存于显存,同时在系统内存中保留FP16原始数据以实现确定性回退。通过两部分误差分解,得到每头每步的注意力分布偏差与值重建误差上界。这些上界在线计算,驱动自适应精度选择与多级回退策略,确保在需要时可完全恢复至精确稠密注意力输出。在LLaMA 3.1-8B模型上,针对最大128K上下文的PG-19、NIAH、RULER基准测试中,该系统在语言建模与检索任务上达到与FP16密集缓存相当的性能,且能修复朴素INT8/INT4基线中的灾难性失效。短上下文下值敏感任务呈现可控压缩-保真权衡,可通过更紧的值容忍度或使用FP16值回退消除。认证为局部(每头每步)性质,不保证端到端模型正确性,但确保每次注意力计算要么在FP16参考下有界,要么通过回退精确恢复。这将量化缓存从固定近似转变为可运行时验证的计算。目标并非单纯加速,而是实现严格质量约束下安全部署激进的缓存压缩。

原文摘要 · Abstract (English)

KV cache quantization reduces the memory cost of long-context LLM inference, but introduces approximation error that is typically validated only empirically. Existing systems rely on average-case robustness, with no mechanism to detect or recover from failures at runtime. We present a tiered KV cache architecture that enables runtime-certified attention: INT8 keys and INT4 values are stored in GPU memory, while FP16 originals are retained in system RAM for deterministic fallback. A two-term error decomposition yields per-head, per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error. These bounds are computed online and used to drive adaptive precision selection and a multi-stage fallback ladder, which guarantees recovery to the exact dense attention output when required. Across PG-19, NIAH, and RULER benchmarks on LLaMA~3.1-8B with contexts up to 128K, the system matches dense FP16 KV quality within noise for language modelling and retrieval tasks, while recovering catastrophic failures observed in naive INT8/INT4 baselines. Value-sensitive tasks at short context expose a controlled trade-off between compression and fidelity, which can be eliminated via tighter value tolerances or FP16-value fallback. The certification is local (per-head, per-step) and does not guarantee end-to-end model correctness, but ensures that each attention computation is either bounded relative to an FP16 reference or exactly recovered via fallback. This reframes KV cache quantization as a runtime-verified computation rather than a fixed approximation. The goal is not raw speedups, but enabling safe deployment of aggressive KV compression under strict quality constraints.

量化注意力长文本可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。