arXiv:2608.01078cs.CLcs.AI2026-08被引 3

通过让模型自省推理过程,实现高效低精度推理大模型量化。

Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization

论文配图:Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
图 1 · 摘自论文原文
  • 用模型自身推理轨迹做校准,突破传统量化瓶颈。
  • 400万校准数据下,17亿参数模型性能达90.52%基准水平。
  • 支持多种架构与任务,极低校准成本,适合部署场景。

我们提出ScaleQ-1.58,一种可扩展的三值后训练量化(PTQ)框架,用于推理类大模型。核心发现是:尽管现代大模型具备链式思维能力,但现有校准方案忽略推理过程,导致即使使用最新基于学习的可微三值化方法(CAT-Q),在数学与编程等挑战性任务上仍出现性能崩溃。为此,我们提出简单校准方法——自思自省(AYOT),将预训练高精度模型在校准样本上的推理轨迹与最终答案作为上下文输入,结合问题共同参与三值化过程。ScaleQ-1.58仅需400万校准令牌,即可使Qwen3-1.7B模型在4个数学与编码任务上达到前人最优BitNet b1.58 2B4T平均性能的90.52%,而其40亿参数版本更实现8.97%绝对提升,且校准令牌需求减少100万倍;该方法适用于密集与MoE架构,随模型规模增大性能持续提升(最高达235B参数);对不同难度任务(数学、编程、科学逻辑、常识推理、基础生成)均有良好泛化能力;性能随校准数据量增加而持续改善。值得注意的是,AYOT在其他位宽量化中也表现出强泛化性。代码将开源。

原文摘要 · Abstract (English)

We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are typically trained to exhibit chain-of-thought reasoning capabilities, in the PTQ regime, even the latest CAT-Q method based on learning-based differentiable ternarization still leads to performance collapse on challenging mathematics and coding tasks when using conventional calibration schemes that ignore the model's reasoning process. Driven by this finding, we introduce a simple calibration approach, Attend to Your Own Thoughts (AYOT), where reasoning traces and final answers generated by the pre-trained high-precision target LLM on a proper set of calibration samples are used as the context input during the ternarization process, along with the corresponding questions. ScaleQ-1.58 is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties: (1) with only 4M calibration tokens, Qwen3-1.7B ternarized by ScaleQ-1.58 reaches over 90.52% of the performance of the prior best BitNet b1.58 2B4T averaged over 4 mathematics and coding tasks, and our ternary Qwen3-4B shows an absolute gain of 8.97%, while requiring 1,000,000x fewer calibration tokens for quantization; (2) ScaleQ-1.58 generalizes well to both dense and MoE architectures, with performance improving as model scale increases (up to 235B parameters); (3) ScaleQ-1.58 demonstrates strong generalization across tasks of varying difficulty levels, including mathematics, coding and scientific logic reasoning, as well as commonsense reasoning and basic language generation; (4) its performance continues to improve as the number of calibration tokens increases. Notably, AYOT also exhibits strong generalization ability across other quantization bit-widths. Code will be available at https://github.com/IntelChina-AI/BitTern.

量化推理大模型低比特

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。