用强化学习自动优化指令，让黑盒大模型表现更优
ACING: Actor-Critic for Instruction Learning in Black-Box LLMs
- 将指令优化建模为无状态连续动作问题，仅靠反馈信号探索无限指令空间
- 在76%任务中超越人工指令，最高提升33分，33项任务中平均提升10分
- 适合需要自动化提示工程的场景，尤其对不可访问参数的模型有效
大型语言模型（LLMs）在完成任务时的表现高度依赖于指令质量，而高质量指令通常需要大量人工设计。这凸显了自动化指令优化的重要性。然而，在黑盒大模型上进行指令优化尤为困难，因为无法获取模型参数和梯度。本文提出ACING，一种基于演员-评论家的强化学习框架，将指令优化建模为无状态、连续动作问题，仅需黑盒反馈即可探索无限指令空间。ACING 在76%的指令归纳任务中优于人工编写的指令，最高提升达33分，且在涵盖指令归纳、摘要和思维链推理的33项任务中，相比最优自动基线实现10分的中位数提升。大量消融实验验证了其鲁棒性与高效性。ACING 的代码已公开于 https://github.com/salmakh1/ACING。
原文摘要 · Abstract (English)
The effectiveness of Large Language Models (LLMs) in solving tasks depends significantly on the quality of their instructions, which often require substantial human effort to craft. This underscores the need for automated instruction optimization. However, optimizing instructions is particularly challenging when working with black-box LLMs, where model parameters and gradients are inaccessible. We introduce ACING, an actor-critic reinforcement learning framework that formulates instruction optimization as a stateless, continuous-action problem, enabling exploration of infinite instruction spaces using only black-box feedback. ACING automatically discovers prompts that outperform human-written prompts in 76% of instruction-induction tasks, with gains of up to 33 points and a 10-point median improvement over the best automatic baseline in 33 tasks spanning instruction-induction, summarization, and chain-of-thought reasoning. Extensive ablations highlight its robustness and efficiency. An implementation of ACING is available at https://github.com/salmakh1/ACING.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。