用强化学习思路优化提示词，让大模型更懂长期对话目标。
Prompt reinforcing for long-term planning of large language models
- 通过生成每轮反馈并重用经验，动态优化提示词实现长期规划。
- 在文本转SQL和任务型对话中显著提升多轮交互表现。
- 无需调整模型参数，适配不同大模型，适合想提升对话能力的研究者。
大语言模型（LLMs）在自然语言处理任务中取得显著进展，可通过提示词进行适配。然而，在多轮交互中仍表现不佳，常依赖错误的早期假设且无法持续追踪用户目标，使这类任务尤为困难。以往对话系统研究显示，长期规划对交互任务至关重要。本文提出一种受强化学习启发的提示优化框架，仅通过修改基于LLM代理的任务指令提示词，即可实现长期规划。该方法通过生成逐轮反馈并利用经验回放进行提示词重写，在文本转SQL和任务导向对话等多轮任务中表现显著提升。此外，该方法可泛化至不同基于LLM的代理，并能使用多种LLM作为元提示生成代理。这为未来基于强化学习的无参数优化方法研究提供了方向。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable success in a wide range of natural language processing tasks and can be adapted through prompting. However, they remain suboptimal in multi-turn interactions, often relying on incorrect early assumptions and failing to track user goals over time, which makes such tasks particularly challenging. Prior works in dialogue systems have shown that long-term planning is essential for handling interactive tasks. In this work, we propose a prompt optimisation framework inspired by reinforcement learning, which enables such planning to take place by only modifying the task instruction prompt of the LLM-based agent. By generating turn-by-turn feedback and leveraging experience replay for prompt rewriting, our proposed method shows significant improvement in multi-turn tasks such as text-to-SQL and task-oriented dialogue. Moreover, it generalises across different LLM-based agents and can leverage diverse LLMs as meta-prompting agents. This warrants future research in reinforcement learning-inspired parameter-free optimisation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。