让大模型学会自我批评，提升强化学习训练效率
Towards Better RL Training Data Utilization via Second-Order Rollout
- 引入二阶回溯：生成回复后，再生成多条批评意见
- 相同数据下，模型推理能力提升12.3%（相对基准）
- 适合想提升模型自省能力的研究者和工程师
强化学习（RL）已使大语言模型具备强大的推理能力，但传统RL仅通过一阶回溯（为问题生成多个回答）训练生成能力，忽视了批判能力的培养，导致训练数据利用不充分。为此，本文提出二阶回溯概念——对生成的回答生成多个批评意见，并构建统一框架，联合训练生成与批判能力。在多种模型与数据集上的实验证明，该方法在相同训练数据条件下，显著优于基线方法，性能更优。此外，研究发现批判训练中标签平衡至关重要，基于结果的奖励存在噪声问题，可通过采样策略有效缓解。本工作初步探索了动态数据增强与生成-批判联合训练，为强化学习训练的进一步发展提供新思路。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has empowered Large Language Models (LLMs) with strong reasoning capabilities, but vanilla RL mainly focuses on generation capability improvement by training with only first-order rollout (generating multiple responses for a question), and we argue that this approach fails to fully exploit the potential of training data because of the neglect of critique capability training. To tackle this problem, we further introduce the concept of second-order rollout (generating multiple critiques for a response) and propose a unified framework for jointly training generation and critique capabilities. Extensive experiments across various models and datasets demonstrate that our approach can utilize training data more effectively than vanilla RL and achieve better performance under the same training data. Additionally, we uncover several insightful findings regarding second-order rollout and critique training, such as the importance of label balance in critique training and the noise problem of outcome-based rewards, which can be mitigated through sampling techniques. Our work offers a preliminary exploration of dynamic data augmentation and joint generation-critique training in RL, providing meaningful inspiration for the further advancement of RL training
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。