提出TEPO框架，解决大模型推理中令牌稀疏奖励问题
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
- 用序列似然将群体奖励关联到单个令牌
- 通过令牌级KL掩码降低熵崩溃风险，提升训练稳定性
- 数学推理任务上效果领先，收敛速度提升50%
组相对策略优化（GRPO）显著提升了大语言模型在数学推理方面的能力。然而，GRPO及相关熵正则化方法仍难以应对链式思维（CoT）推理中的令牌级稀疏奖励问题。这些方法通常采用无差别的令牌级熵正则化，容易在稀疏奖励下导致熵崩溃或模型退化。本文提出TEPO，一种新型的令牌级优化框架：(1) 利用序列级似然将群体级奖励与个体令牌通过令牌级聚合相连接；(2) 引入令牌级KL散度掩码约束，针对具有正优势且熵递减的令牌，缓解突变策略更新。实验表明，TEPO不仅在数学推理基准上达到最先进性能，还显著提升训练稳定性，相比GRPO/DAPO将收敛时间减少50%。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。