量化大模型长文本推理时,新方法解决位置编码失真问题。
Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
- 按频率分组位置编码,动态调整每段缩放系数以稳定输出。
- 在长文本任务上降低困惑度超14%,短文本性能不变。
- 无需微调或修改架构,适合现有部署系统直接使用。
扩展大语言模型的上下文窗口对长距离依赖任务至关重要。基于旋转位置编码(RoPE)的插值与外推方法(如线性缩放和频率感知方案)可在不重新训练的情况下支持更长输入,而训练后量化(PTQ)使部署成为可能。然而我们发现,将RoPE插值(PI)与PTQ结合会因多重耦合效应导致精度下降,包括长上下文混淆、动态范围膨胀、轴对齐量化器与旋转位置编码对之间的各向异性,以及异常值偏移带来的位置相关逻辑噪声。本文首次系统分析了PI+PTQ方法,提出两个实用诊断工具:插值压力(每频段对相位缩放的敏感度)和尾部膨胀比(短到长上下文中的异常值偏移)。基于分析结果,我们提出Q-ROAR(量化、位置插值、异常值感知重缩放)——一种仅调整权重、适应插值的稳定化方法。Q-ROAR将RoPE维度划分为少量频率带,对键(Key)和查询(Query)权重进行轻量级每带尺度搜索(可选对称变体以保持逻辑尺度)。搜索由诊断引导,使用极小的长上下文开发数据集,无需模型微调、架构或内核改动,也无额外部署开销。实验证明,Q-ROAR在长上下文任务上将困惑度降低超过14%,同时保持短上下文性能、推理吞吐量和与现有LLM系统栈的兼容性。
原文摘要 · Abstract (English)
Extending the context window support of large language models (LLMs) is crucial for tasks with long-distance dependencies. RoPE-based interpolation and extrapolation methods, such as linear scaling and frequency-aware schemes, enable longer input length support without retraining, while post-training quantization (PTQ) makes deployment practical. However, we show that combining RoPE position interpolation (PI) with PTQ degrades accuracy due to coupled effects including long-context aliasing, dynamic-range dilation, anisotropy from axis-aligned quantizers vs. rotated RoPE pairs, and outlier shifting that produces position-dependent logit noise. We provide, to the best of our knowledge, the first systematic analysis of the PI+PTQ approach and introduce two practical diagnostics: interpolation pressure (per-band sensitivity to phase scaling) and tail-inflation ratios (outlier shift from short to long contexts). Following the analysis results, we propose Q-ROAR (Quantization, RoPE-interpolation, and Outlier Aware Rescaling), a weight-only, interpolation-aware stabilization of PI for quantized LLMs. Q-ROAR groups RoPE dimensions into a small number of frequency bands and performs a lightweight search over per-band scales for Key and Query weights (with an optional symmetric variant to preserve logit scale). The search is guided by our diagnostics and uses a tiny long-context development dataset, requiring no fine-tuning to the model, no architecture or kernel changes, and no additional deployment overhead. Empirically, Q-ROAR reduces the model's perplexity on long-context workloads by more than 14%, while preserving short-context performance, inference throughput, and compatibility with existing LLM system stacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。