用强化学习自动优化提示词，让大模型更稳定高效。
StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models
- 将提示词调优建模为在线强化学习问题，引入自适应更新策略。
- 在文本分类、问答和生成任务上均优于现有方法。
- 适合需要高效提示工程的AI研发人员使用。
随着大语言模型（LLM）应用的扩展，为特定任务找到合适的提示词变得至关重要。尽管强化学习（RL）广泛用于提示词调优，但其固有的不稳定性与环境依赖性使其难以实际应用。本文提出StablePrompt，平衡训练稳定性与搜索空间，缓解了强化学习的不稳定性，并生成高性能提示词。我们将提示词调优建模为代理与目标LLM之间的在线强化学习问题，并引入自适应近端策略优化（APPO）。APPO通过引入一个LLM锚定模型，自适应调整策略更新速率，实现灵活的提示词搜索，同时保留预训练LLM的语言能力。StablePrompt在文本分类、问答和文本生成等多种任务上均超越了先前方法。代码已开源。
原文摘要 · Abstract (English)
Finding appropriate prompts for the specific task has become an important issue as the usage of Large Language Models (LLM) has expanded. Reinforcement Learning (RL) is widely used for prompt tuning, but its inherent instability and environmental dependency make it difficult to use in practice. In this paper, we propose StablePrompt, which strikes a balance between training stability and search space, mitigating the instability of RL and producing high-performance prompts. We formulate prompt tuning as an online RL problem between the agent and target LLM and introduce Adaptive Proximal Policy Optimization (APPO). APPO introduces an LLM anchor model to adaptively adjust the rate of policy updates. This allows for flexible prompt search while preserving the linguistic ability of the pre-trained LLM. StablePrompt outperforms previous methods on various tasks including text classification, question answering, and text generation. Our code can be found in github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。