将强化学习与提示优化结合，提升语言模型程序的准确性。
Composing Policy Gradients and Prompt Optimization for Language Model Programs
- 以模块调用为单位构建分组，实现多提示程序的GRPO算法。
- 在分类、多跳搜索等任务上平均提升11%准确率，优于单独使用提示优化。
- 开源于DSPy库，适合需要优化复杂语言模型程序的研究者。
组相对策略优化（GRPO）已被证明是微调语言模型的有效方法。然而，当前AI系统越来越多地表现为混合多个语言模型调用、不同提示模板及其他工具的模块化程序，如何有效利用在线强化学习算法如GRPO来改进这类系统仍不明确。本文首次探讨了将GRPO应用于任意多提示程序的可行性，并验证其作为语言模型程序的即插即用优化器的鲁棒性。我们提出的多模块GRPO以模块级调用为分组依据，也考察了轨迹级分组的自然实现方式。实验表明，GRPO及其多模块版本能有效与自动提示优化协同工作，在分类、多跳搜索和隐私保护委托任务中，相比后训练语言模型平均提升11%准确率，较仅使用提示优化再提高5%。相关代码已开源至DSPy库（https://dspy.ai）。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has proven to be an effective tool for post-training language models (LMs). However, AI systems are increasingly expressed as modular programs that mix together multiple LM calls with distinct prompt templates and other tools, and it is not clear how practitioners can best leverage online RL algorithms like GRPO to improve these systems. We begin to address this challenge by investigating whether it is possible to effectively instantiate GRPO for arbitrary multi-prompt programs and whether it can work robustly as an off-the-shelf optimizer for LM programs using the same abstractions and constraints typically involved for prompt optimization. Our main variant of multi-module GRPO constructs groups from module-level invocations, and we also consider trajectory-level grouping as another natural instantiation. We find for the first time that GRPO (and its multi-module counterpart) empirically composes well with automatic prompt optimization, and together they improve accuracy by 11% on average across classification, many-hop search, and privacy-preserving delegation tasks against the post-trained LM - with 5% gains against prompt optimization on its own. We open-source multi-module GRPO in the DSPy library at https://dspy.ai .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。