改进量化校准，让模型更适应不同长度输入。
On the Importance of a Multi-Scale Calibration for Quantization
- 用多尺度输入长度构建更准确的权重重要性评估。
- 在低比特量化下，多个主流大模型精度显著提升。
- 无需改动现有框架，适合想优化量化效果的研究者。
后训练量化（PTQ）是高效部署大语言模型（LLMs）的核心技术，其性能高度依赖于小规模校准集。然而，传统方法采用固定长度的随机序列，忽略了LLM输入的可变长度特性。输入长度直接影响激活分布，进而影响通过海瑟矩阵（Hessian）捕捉的权重重要性，导致固定长度校准得到的海瑟矩阵难以反映多样输入场景下的真实权重重要性。为此，我们提出MaCa（Matryoshka Calibration），一种简单有效的长度感知海瑟构造方法：（i）将多尺度序列长度信息融入海瑟估计；（ii）将每条序列视为独立样本进行正则化，从而获得更稳定、更有效的海瑟矩阵，实现精准量化。在Qwen3、Gemma3、LLaMA3等先进LLMs上的实验表明，MaCa在低比特量化下持续提升精度，是一种轻量级且兼容现有PTQ框架的改进方案。据我们所知，这是首个系统揭示多尺度校准在LLM量化中作用的工作。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) is a cornerstone for efficiently deploying large language models (LLMs), where a small calibration set critically affects quantization performance. However, conventional practices rely on random sequences of fixed length, overlooking the variable-length nature of LLM inputs. Input length directly influences the activation distribution and, consequently, the weight importance captured by the Hessian, which in turn affects quantization outcomes. As a result, Hessian estimates derived from fixed-length calibration may fail to represent the true importance of weights across diverse input scenarios. We propose MaCa (Matryoshka Calibration), a simple yet effective method for length-aware Hessian construction. MaCa (i) incorporates multi-scale sequence length information into Hessian estimation and (ii) regularizes each sequence as an independent sample, yielding a more stable and fruitful Hessian for accurate quantization. Experiments on state-of-the-art LLMs (e.g., Qwen3, Gemma3, LLaMA3) demonstrate that MaCa consistently improves accuracy under low bit quantization, offering a lightweight enhancement compatible with existing PTQ frameworks. To the best of our knowledge, this is the first work to systematically highlight the role of multi-scale calibration in LLM quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。