通过优化部分未来奖励，提升LLM强化学习中策略学习的可靠性。
Partial Policy Gradients for RL in LLMs
- 用子集未来奖励替代全序列奖励，简化策略结构
- 不同策略在对话任务中表现各异，验证了策略类别的差异性
- 适合研究策略设计与个性化对话的RL应用
强化学习是一种在未知环境中顺序决策的学习框架。我们提出一种自然的方法来建模策略梯度中的策略结构：优化未来奖励的一个子集。较小的子集对应更简单的策略，因其经验梯度估计更准确，可更可靠地学习。该方法支持对多种策略类别的建模与比较，包括完整规划、贪心策略、K步前瞻和分段策略。我们在多个个性对齐的对话任务上进行了实证评估，结果表明不同策略在不同问题上表现优异，反映了其各自特性，凸显了所研究策略类别的重要性。
原文摘要 · Abstract (English)
Reinforcement learning is a framework for learning to act sequentially in an unknown environment. We propose a natural approach for modeling policy structure in policy gradients. The key idea is to optimize for a subset of future rewards: smaller subsets represent simpler policies, which can be learned more reliably because their empirical gradient estimates are more accurate. Our approach allows for modeling and comparison of different policy classes, including full planning, greedy, K-step lookahead, and segment policies. We evaluate the policies empirically on multiple persona-alignment conversational problems. Different policies excel in different problems, reflecting their different characteristics and highlighting the importance of our studied policy class.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。