用更精准的约束机制替代PPO剪裁，提升大模型强化学习稳定性与效果。
TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
- 引入可微分的离散信任区域投影，实现逐标记KL约束。
- 在数学推理与代码生成任务中，训练速度更快、成功率更高。
- 兼容多种模型和优势估计方法，适合追求稳定训练的大模型应用。
基于奖励的大语言模型微调普遍采用类似PPO的剪裁目标，尽管近期研究改进了优势估计与归一化方法，但剪裁机制本身仍沿用旧法。原始剪裁作为基于KL的信任区域的粗糙近似，常导致更新不稳定和性能不佳。本文提出一种新的可微分离散信任区域投影，对模型关键标记的输出进行精确的令牌级KL约束，同时仅处理稀疏子集以平衡计算成本与有效性。所提方法TROLL可直接替换训练中的PPO式剪裁，不改变推理行为。在数学推理和代码生成任务中，涵盖不同模型族与优势估计方法，TROLL在训练速度、稳定性和最终成功率上均持续优于传统剪裁方法。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) with PPO-like clip objectives has become the standard choice for reward-based fine-tuning of large language models (LLMs). Although recent work has explored improved estimators of advantages and normalization, the clipping mechanism itself has remained untouched. Originally introduced as a proxy for principled KL-based trust regions, clipping is a crude approximation that often causes unstable updates and suboptimal performance. We replace the clip objective with a novel discrete differentiable trust region projection, which provides principled token-level KL constraints. The projection operates on a sparse subset of the model's most important token logits to balance computational cost and projection effectiveness. Our approach, Trust Region Optimization for Large Language models (TROLL), serves as a direct replacement for PPO-like clipping during training and does not alter the model's inference behavior. Across mathematical reasoning and code generation tasks, model families, as well as advantage-estimation methods, TROLL consistently outperforms PPO-like clipping in terms of training speed, stability, and final success rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。