通过群体感知强化学习提升大模型输出多样性
Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
- 基于群体整体表现设计奖励机制，引导模型生成更丰富回答
- 在GSM8K等基准上保持准确率的同时，显著提升响应多样性
- 适用于开放性任务，适合需要多角度生成的场景
大语言模型常出现模式崩溃问题，即在存在多种合理答案时反复生成少数几种结果，限制了其在各类任务中的多样性。本文提出群体感知策略优化（GAPO），作为近期流行的组相对策略优化（GRPO）的简单扩展，通过整体群体层面计算奖励，使模型能学习多样性与覆盖度等群体属性。我们采用频率感知奖励函数，鼓励对有效生成结果进行均匀采样，实验表明经GAPO训练的模型能生成更多样且有效的回答。该方法还可推广至开放式提示，在不降低GSM8K、MATH、HumanEval、MMLU-Pro等标准基准准确率的前提下，有效提升响应多样性。代码将公开发布。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。