用对抗生成策略让大模型更好理解精细指令约束。
GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization
- 通过对抗训练自动生成不同难度的训练样本
- 在多个基准上优于PPO、DPO等方法，尤其擅长细粒度约束
- 适合需要精准控制输出的场景，如合规文本生成
大语言模型对输出控制的需求日益迫切，现有方法在处理细粒度约束时常因理解偏差导致幻觉或性能脆弱。我们提出生成对抗策略优化（GAPO），结合GAN训练机制与仅编码器结构的奖励模型，逐步学习并适应复杂约束。GAPO利用对抗训练自动生成不同难度的训练样本，并通过编码器架构更准确捕捉提示-响应关系。大量实验表明，该方法在多个基准测试中表现优异，尤其在细粒度约束任务中显著超越PPO、DPO和KTO。结果证明GAPO为大模型输出控制提供了更鲁棒高效的解决方案。代码已开源：https://github.com/MikeGu721/GAPO。
原文摘要 · Abstract (English)
Recent advances in large language models have highlighted the critical need for precise control over model outputs through predefined constraints. While existing methods attempt to achieve this through either direct instruction-response synthesis or preferential response optimization, they often struggle with constraint understanding and adaptation. This limitation becomes particularly evident when handling fine-grained constraints, leading to either hallucination or brittle performance. We introduce Generative Adversarial Policy Optimization (GAPO), a novel framework that combines GAN-based training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. GAPO leverages adversarial training to automatically generate training samples of varying difficulty while utilizing the encoder-only architecture to better capture prompt-response relationships. Extensive experiments demonstrate GAPO's superior performance across multiple benchmarks, particularly in scenarios requiring fine-grained constraint handling, where it significantly outperforms existing methods like PPO, DPO, and KTO. Our results suggest that GAPO's unique approach to preferential prompt learning offers a more robust and effective solution for controlling LLM outputs. Code is avaliable in https://github.com/MikeGu721/GAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。