arXiv:2605.18529cs.AI2026-05

提出新方法提升大模型推理时的逐标记信用分配精度。

AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment

论文配图:AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
图 1 · 摘自论文原文
  • 用自我反思瓶颈压缩验证反馈生成精炼提示,避免直接暴露答案
  • 通过因果信息增益实现稀疏精准的令牌级奖励调制,提升稳定性
  • 适用于科学、数学等长链推理任务,防止训练后期崩溃

大语言模型在复杂推理中的对齐高度依赖可验证奖励的强化学习(RLVR)。然而,标准算法如GRPO对所有令牌统一施加序列级奖励,导致严重的信用分配瓶颈。虽然在线策略自蒸馏尝试通过将自教师条件化于特权上下文来缓解问题,但直接暴露于原始参考轨迹常引发教师分布过度条件化、隐式答案泄露及后期训练崩溃。为此,本文提出非对称元反思自蒸馏(AMR-SD):不直接依赖原始参考轨迹,而是通过反射瓶颈将验证结果、同伴采样或参考反馈等诊断信号压缩为简洁的自我生成苏格拉底式提示与批判。进一步引入因果信息增益(CIG),采用非对称ReLU门控阈值,将这些反思转化为稀疏且高精度的令牌级优势调制。结合时间退火机制,在保留基础环境奖励的同时过滤分布噪声。跨科学、数学及工具使用基准的实验表明,AMR-SD显著优于现有基线,实现稳健的长时程稳定性,并成功防止晚期训练崩溃。

原文摘要 · Abstract (English)

The alignment of Large Language Models (LLMs) for complex reasoning heavily relies on Reinforcement Learning with Verifiable Rewards (RLVR). However, standard algorithms like GRPO apply sequence-level rewards uniformly to all tokens, creating a severe credit-assignment bottleneck. While on-policy self-distillation attempts to resolve this by conditioning a self-teacher on privileged contexts, direct exposure to raw oracle solutions often induces over-conditioned teacher distributions, implicit answer leakage, and late-stage training collapse. To overcome these limitations, we propose Asymmetric Meta-Reflective Self-Distillation (AMR-SD). Instead of conditioning directly on raw reference traces, AMR-SD inserts a reflection bottleneck: it compresses diagnostic signals -- from verifier outcomes, peer rollouts, or reference feedback -- into concise, self-generated Socratic hints and critiques. Furthermore, we introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations. Combined with temporal annealing, this mechanism preserves the base environmental reward while filtering out distributional noise. Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines, achieving robust long-horizon stability and successfully preventing late-stage collapse.

大模型推理信用分配自蒸馏强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。