让模型自动生成奖励，实现高效前瞻生成。
Streaming Looking Ahead with Token-level Self-reward
- 用自监督奖励模型替代外部奖励机制，实现无需通信的前瞻生成。
- 在三个数据集上胜率79.7%，结合强化微调可提升至89.4%。
- 专为流式场景设计，兼顾推理效率与输出质量，适合实时生成任务。
自回归解码通常仅依赖历史信息，难以达到最优性能。近期研究发现，借助外部奖励模型（RMs）的前瞻算法（如蒙特卡洛树搜索，MCTS）可通过预判未来输出和奖励来优化当前生成，显著提升强化微调与推理效果。然而其高计算开销限制了在流式场景中的应用。为此，我们提出在策略模型中引入词元级自奖励建模（TRM），构建无需外部模型与通信的奖励转换器（Reward Transformer）。同时设计流式前瞻（SLA）算法，通过更好并行化提升搜索效率。实验表明，使用冻结策略模型时，SLA在三个通用领域数据集上对基线贪婪解码的总体胜率达79.7%；若结合如DPO等强化微调技术，胜率进一步提升至89.4%。
原文摘要 · Abstract (English)
Autoregressive decoding algorithms that use only past information often cannot guarantee the best performance. Recently, people discovered that looking-ahead algorithms such as Monte Carlo Tree Search (MCTS) with external reward models (RMs) can significantly improve models' output by allowing them to think ahead and leverage future outputs and associated rewards to guide the current generation. Such techniques can help the reinforcement fine-tuning phase by sampling better trajectories and the inference phase by selecting the better output. However, their high computational cost limits their applications, especially in streaming scenarios. To address this issue, we propose equipping the policy model with token-level self-reward modeling (TRM) capability to eliminate the need for external models and extra communication. We name the new architecture as Reward Transformer. In addition, we propose a streaming-looking-ahead (SLA) algorithm to further boost search efficiency with better parallelization. Experiments show that SLA achieves an overall win rate of 79.7\% against the baseline greedy decoding algorithm on three general-domain datasets with a frozen policy model while maintaining streaming efficiency. If we combine SLA with reinforcement fine-tuning techniques such as DPO, SLA achieves an overall win rate of 89.4\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。