通过遮蔽边界暂停标记，提升大模型推理能力并保留语言理解。
Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective
- 在推理步骤边界设置遮蔽暂停标记，调控训练动态
- 数学任务最高提升6分，代码任务提升2.5分，保持语言能力
- 适用于需要推理增强且不牺牲通用能力的场景
暂停标记通过在序列中插入特殊标记来提升大模型的推理能力。以往研究多从计算表达性角度解释其效果，但对暂停标记的训练动态研究较少。本文通过两个受控实验揭示显著不对称性：在合成持续学习任务中，掩码暂停标记在相同最终适应程度下，对先前学习分布的覆盖仅约为未掩码情况的1/4（模式保留）；在合成数学推理探测任务中，边界邻近标记能编码更多下游步骤信息（非短视压缩）。为此提出一种训练规则——掩码边界暂停（MBP），即在推理步骤边界放置暂停标记并屏蔽其损失。在1B-8B规模的Qwen与Llama模型上，MBP一致提升推理表现，数学任务最高提升6分，代码任务提升2.5分，同时保持通用语言理解能力。进一步验证该策略可扩展至GRPO。结果表明，暂停标记本质是干预训练动态中保留与适应的权衡，而非仅限于推理阶段的计算工具。
原文摘要 · Abstract (English)
Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4x less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent token comes to encode substantially more downstream-step information (H2, non-myopic compression). We formalize a training rule consistent with both - Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked. Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code, while preserving general language understanding abilities. We further demonstrate that this mode-preserving strategy extend gains to GRPO. These results recast pause tokens as a training-dynamics intervention on the retention-adaptation trade-off, rather than merely an inference-time computation device.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。