用排序奖励引导强化学习，让模型更按标准步骤解题。
Bootstrapped Mixed Rewards for RL Post-Training: Injecting Canonical Action Order
- 用结构化排序信号辅助强化学习后训练。
- 在泽比拉谜题上性能优于仅用任务奖励的模型。
- 适合需要逻辑顺序的推理任务研究者。
后训练阶段的强化学习（RL）通常只优化单一标量目标，忽略了求解过程中的结构信息。本文提出在仅用于强化学习后训练时，引入一个关于标准求解顺序的粗粒度提示信号，以提升模型性能，即使在随机解题序列上进行微调也有效。在泽比拉谜题任务中，先在随机解题顺序上微调Transformer模型，再使用组相对策略优化（GRPO）进行后训练，采用两种奖励：仅当谜题完全解决时为1的稀疏任务奖励，以及随着模型输出顺序与标准求解顺序一致而增加的排序奖励。通过固定混合比例并使用简单的自举缩放方法，在初始化时均衡各成分大小，实验表明混合奖励普遍优于仅使用任务奖励的情况，说明粗粒度的排序信号可在不修改监督数据或架构的前提下，有效引导强化学习后训练朝向标准解题路径。
原文摘要 · Abstract (English)
Post-training with reinforcement learning (RL) typically optimizes a single scalar objective and ignores structure in how solutions are produced. We ask whether a scalar hint toward a canonical solver ordering, used only during RL post-training, improves performance even when fine-tuned on randomized solution sequences. On Zebra puzzles, we fine-tune a Transformer on randomized solution orders, then post-train it with Group Relative Policy Optimization (GRPO) using two rewards: a sparse task reward that is 1 only when the puzzle is fully solved, and an ordering reward that increases when the model's emission order aligns with the canonical solver order. To compare signals cleanly, we combine them via fixed mixtures and use a simple bootstrapped scaling to equalize component magnitudes at initialization. Mixed rewards generally outperform task-only optimization, suggesting that coarse ordering signals can steer RL post-training toward canonical trajectories without modifying supervised data or architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。