arXiv:2605.11609cs.LGcs.AI2026-05被引 10

让模型自省推理,用反自蒸馏提升数学解题能力

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

论文配图:Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
图 1 · 摘自论文原文
  • 通过反转自蒸馏方向,让模型更关注推理过程而非已有结论
  • 在5个4B-30B模型上,训练步数减少2到10倍,准确率最高提升11.5点
  • 适合追求高效自进化推理的AI研究者和开发者

在策略自蒸馏中,学生模型被拉向一个带有特权上下文(如验证解或反馈)的自身副本,为提升推理能力提供了新路径。然而在数学推理任务中,效果不稳定,尽管同一方法在其他场景成功。通过点互信息分析发现:特权上下文会夸大教师对已隐含内容(如结构连接词、可验证命题)的信心,同时削弱对推理关键词(如“等待”、“让我们”、“也许”)的关注,这些词正是多步搜索的核心。为此提出反自蒸馏(AntiSD),不缩小学生与教师间的差异,而是扩大它;此操作反转了每个词的梯度符号,一步内实现自然有界的奖励。通过熵触发门控机制,在教师熵坍缩时自动关闭该项,实现对标准自蒸馏的即插即用替换。在五个4B至30B参数规模的模型上,抗自蒸馏在2至10倍更少的训练步数内达到GRPO基线精度,并将最终准确率提升最多11.5分。该方法为语言模型通过自身训练信号实现可扩展的自我改进开辟了道路。

原文摘要 · Abstract (English)

On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.

自蒸馏推理增强强化学习数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。