通过熵权重细化奖励分配，让大模型推理更精准。
GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
- 用令牌和序列级熵权重重新分配奖励，实现细粒度反馈。
- 在长链思维任务中，新方法使推理准确率提升12.7%。
- 适合需要精准推理的AI系统，如数学解题与逻辑推演。
强化学习在提升大语言模型推理能力中至关重要，但主流算法如GRPO和DAPO受限于粗粒度信用分配机制，同一响应中的所有标记获得相同奖励。本文提出动态熵加权，系统定义基于熵的权重比 $\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$ 及其变体，通过两种新算法实现细粒度奖励分配：组令牌策略优化（GTPO）为每个令牌分配熵加权奖励，并构建特定令牌的优势函数以引导模型走向最优路径；序列级GRPO（GRPO-S）将该设计扩展至序列层级，在长链思维（CoT）推理任务中表现出更优稳定性。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all tokens within the same response receive the identical reward. In this paper, we propose Dynamic Entropy Weighting, systematically define entropy-based weight ratios $\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$ and similar variants to redistribute rewards and get fine-grained rewards through two new algorithms: Group Token Policy Optimization (GTPO), which assigns an entropy-weighted reward to each token and synthesizes token-specific advantage function to drive the model toward optimal path, and the analogous algorithm Sequence-Level GRPO (GRPO-S), which extends this design to the sequence level and exhibits superior stability in long Chain-of-Thought (CoT) reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。