用符号对齐方法实现无需训练的推理知识蒸馏，提升学生模型逻辑一致性。
SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning
- 将自然语言推理过程转化为符号里程碑，实现序列级对齐。
- 通过最长公共子序列提供密集且有序的奖励信号，效果优于稀疏奖励。
- 无需神经验证器，可直接用于数学、科学等领域的推理任务。
基于强化学习的知识蒸馏有望将复杂推理从教师模型迁移到学生模型，但面临关键困境：要么使用稀疏的结果奖励（缺乏逻辑引导），要么依赖昂贵的神经过程奖励模型（PRM）获取密集信号。本文提出SPEAR（符号过程评估与对齐奖励），一种无需训练、即插即用的序列级在线策略蒸馏过程奖励方法。SPEAR将自然语言推理轨迹投影为领域自适应的符号里程碑，以高效代理过程级推理对齐。通过最长公共子序列（LCS）对齐学生探索路径与教师里程碑，SPEAR提供密集且顺序敏感的奖励信号，无需外部神经验证器即可强制逻辑一致性。在数学、科学和常识推理任务上的实验表明，SPEAR通过高效的密集过程奖励，有效缩小了师生模型间的推理差距。代码与数据已公开于：https://github.com/zhuochunli/SPEAR。
原文摘要 · Abstract (English)
Reinforcement learning-based knowledge distillation has the potential to transfer complex reasoning from teacher to student models, yet it currently faces a critical dilemma: researchers must choose between sparse outcome-based rewards, which provide insufficient logical guidance, or expensive neural Process Reward Models (PRMs) for dense signals. We resolve this by introducing SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation. SPEAR projects natural-language reasoning traces into domain-adaptive symbolic milestones, providing an efficient proxy for process-level reasoning alignment. By utilizing the longest common subsequence (LCS) to align student explorations with teacher milestones, SPEAR provides a dense, order-aware reward signal that enforces logical consistency without the need for an external neural verifier. Our experiments across math, science, and commonsense reasoning tasks demonstrate that SPEAR effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewards. Our code and data are available at: https://github.com/zhuochunli/SPEAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。