不训练直接循环模型层，提升大模型推理性能
Training-Free Looped Transformers
- 测试时用轻量包装循环模型中间层，无需微调或改结构
- 在多个模型上提升2.64、1.14、1.20个百分点准确率
- 适合想提升现有模型性能又不想重新训练的用户
我们提出训练自由的循环变压器，通过在推理时对冻结检查点的连续中间层块添加轻量级包装实现循环，无需额外微调、继续训练或架构修改。不同于以往需端到端训练的循环方法，我们仅在测试时为预训练模型引入递归机制。实验表明，简单重复块通常会降低性能，凸显循环策略的重要性。受预归一化变换块可视为常微分方程前向欧拉步的启发，我们将循环视为对同一近似的改进，用更小的阻尼子步替代一次大更新。在七种密集、稀疏MoE及MLA+MoE模型族中，该方法使Qwen3-4B-Instruct在MMLU-Pro上提升2.64个百分点，Qwen3-30B-A3B-Instruct在CommonsenseQA上提升1.14个百分点，Moonlight-16B-A3B-Instruct在OpenBookQA上提升1.20个百分点。
原文摘要 · Abstract (English)
We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm transformer block as a forward Euler step on an ODE, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our method improves Qwen3-4B-Instruct by +2.64 pp on MMLU-Pro, Qwen3-30B-A3B-Instruct by +1.14 pp on CommonsenseQA, and Moonlight-16B-A3B-Instruct by +1.20 pp on OpenBookQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。