用程序化奖励直接训练大模型做数学编程，效果有限但有新方法。
Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards
- 用程序化奖励替代人工反馈，直接训练大模型完成形式语言任务。
- 简单算术任务成功率低，复杂游戏生成任务几乎失败。
- 提出批量熵正则化提升探索能力，适合微调对齐而非学新任务。
近端策略优化（PPO）常用于从人类反馈中强化学习以对齐大语言模型（LLM）与下游任务。本文探讨了使用PPO进行直接强化学习（RL）的可行性，即通过显式编程的奖励信号，而非依赖中间奖励模型的人类反馈。研究聚焦于数学、编程等形式语言任务，这些任务可通过程序化奖励自动评估生成结果质量。实验涵盖情感对齐任务、简单算术任务和更复杂的博弈合成任务。情感对齐任务复现已有研究以验证实验设置；结果显示，纯基于强化学习的训练在两个形式语言任务中均具挑战性，即使在简单算术任务上成功也受限。为此，我们提出一种新型批量熵正则化项以促进探索，但训练仍未完全稳定。研究发现，直接强化学习更适合微小调整如对齐任务，而非从零学习新任务，即使可编程奖励信号信息量充足。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) is commonly used in Reinforcement Learning from Human Feedback to align large language models (LLMs) with downstream tasks. This paper investigates the feasibility of using PPO for direct reinforcement learning (RL) from explicitly programmed reward signals, as opposed to indirect learning from human feedback via an intermediary reward model. We focus on tasks expressed through formal languages, such as mathematics and programming, where explicit reward functions can be programmed to automatically assess the quality of generated outputs. We apply this approach to a sentiment alignment task, a simple arithmetic task, and a more complex game synthesis task. The sentiment alignment task replicates prior research and serves to validate our experimental setup. Our results show that pure RL-based training for the two formal language tasks is challenging, with success being limited even for the simple arithmetic task. We propose a novel batch-entropy regularization term to aid exploration, although training is not yet entirely stable. Our findings suggest that direct RL training of LLMs may be more suitable for relatively minor changes, such as alignment, than for learning new tasks altogether, even if an informative reward signal can be expressed programmatically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。