量化长上下文大模型时，新方法修复位置编码插值的误差问题。
Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs
- 按频率分组位置编码维度，动态调整权重缩放以缓解误差
- 在标准任务上恢复0.7%准确率，报告数据困惑度降低超10%
- 无需微调或修改架构，适合现有推理系统部署
扩展大模型上下文窗口对长序列任务至关重要。基于RoPE的位置插值（PI）方法如线性与频域感知缩放可在不重新训练的情况下延长输入长度，而后训练量化（PTQ）则实现实际部署。我们发现，将PI与PTQ结合会因长上下文混叠、动态范围扩张、轴向网格各向异性及异常值偏移等耦合效应导致位置相关的逻辑值噪声，显著降低精度。本文首次系统分析了该问题，提出两个诊断指标：插值压力（每频段相位缩放敏感度）与尾部膨胀比（短/长上下文异常值偏移量）。为此，我们提出Q-ROAR：一种面向RoPE的权重仅稳定机制，将RoPE维度分组为若干频段，对每个频段搜索最优缩放系数应用于W_Q、W_K，可选对称变体以保持逻辑值尺度。诊断引导的搜索仅需极小的长上下文开发集，无需微调、内核或架构变更。实验证明，Q-ROAR在标准任务上恢复最高0.7%准确率，政府报告困惑度降低超过10%，同时保持短上下文性能和与现有推理栈兼容性。
原文摘要 · Abstract (English)
Extending LLM context windows is crucial for long range tasks. RoPE-based position interpolation (PI) methods like linear and frequency-aware scaling extend input lengths without retraining, while post-training quantization (PTQ) enables practical deployment. We show that combining PI with PTQ degrades accuracy due to coupled effects long context aliasing, dynamic range dilation, axis grid anisotropy, and outlier shifting that induce position-dependent logit noise. We provide the first systematic analysis of PI plus PTQ and introduce two diagnostics: Interpolation Pressure (per-band phase scaling sensitivity) and Tail Inflation Ratios (outlier shift from short to long contexts). To address this, we propose Q-ROAR, a RoPE-aware, weight-only stabilization that groups RoPE dimensions into a few frequency bands and performs a small search over per-band scales for W_Q,W_K, with an optional symmetric variant to preserve logit scale. The diagnostics guided search uses a tiny long-context dev set and requires no fine-tuning, kernel, or architecture changes. Empirically, Q-ROAR recovers up to 0.7% accuracy on standard tasks and reduces GovReport perplexity by more than 10%, while preserving short-context performance and compatibility with existing inference stacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。