针对递归变压器量化难题，提出新方法显著提升精度与稳定性。
LoopQ: Quantization for Recursive Transformers
- 设计环路感知量化框架，动态调整激活与状态对齐
- 在W4A4量化下平均准确率提升68.8%，困惑度降低87.7%
- 适合关注模型压缩与高效推理的研究者和工程师
循环语言模型（LoopLM）通过递归复用Transformer模块，在固定模型规模下实现更深计算，但其复用机制使模型在训练后量化（PTQ）下更脆弱。本文首次系统研究了LoopLM中的量化问题，识别出三大挑战：角色间分布偏移、环路间状态复用及递归误差累积。为此提出LoopQ，一种环路感知的PTQ框架，保留共享量化主干的同时引入轻量适配。LoopQ结合激活缩放、选择性变换、跨环状态对齐与轨迹感知优化，有效缓解环内分布不匹配与环间误差累积。在七个基准测试中，相比最强静态PTQ基线，于W4A4量化下平均下游准确率提升68.8%，平均困惑度降低87.7%。
原文摘要 · Abstract (English)
Looped language models (LoopLMs) improve parameter efficiency by recursively reusing Transformer blocks, enabling deeper computation under a fixed model size. However, this reuse makes LoopLMs more fragile under post-training quantization (PTQ). We present the first systematic study of quantization in LoopLMs and identify three challenges: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. To address these challenges, we propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines activation scaling, selective transformation, cross-loop state alignment, and trajectory-aware optimization to reduce distributional mismatch within loops and error accumulation across loops. Experiments across seven benchmarks show that, under W4A4 quantization, LoopQ improves average downstream accuracy by 68.8% and reduces average perplexity by 87.7% compared with the strongest static PTQ baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。