通过自适应遮蔽与插值优化推理模型训练，提升数学解题能力。
PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners
- 基于推理路径与答案重叠度动态遮蔽部分正确答案，引导模型关注关键步骤
- 在熵不匹配的少数位置进行能量空间插值，增强训练信号密度
- 在多个Qwen3模型上显著提升数学竞赛题准确率，适合强化推理训练场景
提升大语言模型推理能力需要既与模型测试时状态对齐、又在词元层面提供信息的监督。强化学习使用可验证奖励实现在线探索，但信用分配稀疏且方差高；监督微调与蒸馏提供密集目标，但常依赖固定轨迹或更强教师模型。近期的特权在线自蒸馏通过在已验证解上下文中用同一模型评分学生推演路径，探索了中间路线。本文从上下文重评分视角重新审视该设定：推理中不仅需判断特权上下文是否存在，还需确定其揭示程度及分布如何影响学生学习。提出PAINT（部分解自适应插值训练），根据推演路径与参考答案的重叠度遮蔽验证解，并在熵不匹配的少量词元位置进行小规模能量空间插值。在多个竞赛级数学基准上，PAINT在所有三个Qwen3规模下均持续优于强基线。在Qwen3-8B上，相较之前自蒸馏基线提升宏平均准确率2.1点，较GRPO提升2.9点。
原文摘要 · Abstract (English)
Improving large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level. Reinforcement learning with verifiable rewards provides on-policy exploration but offers sparse, high-variance credit; supervised fine-tuning and distillation provide dense targets but often train on fixed trajectories or rely on stronger teachers. Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context. We revisit this setting through a contextual re-scoring lens: for reasoning, the important choices are not only whether privileged context is available, but how much of it should be revealed and where its distribution should shape the student. We propose PAINT (Partial-solution Adaptive INterpolated Training), which masks the verified solution according to rollout-reference overlap and applies a small energy-space interpolation on a sparse set of entropy-mismatch token positions. Across competition-level math benchmarks, PAINT consistently improves over a strong prior on-policy self-distillation baseline at all three Qwen3 scales. On Qwen3-8B, it raises macro Avg@12 by 2.1 points over this prior baseline and 2.9 points over GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。