用文本反馈训练专用提示模型，提升大模型推理能力。
TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards
- 用文本奖励直接优化提示模型，无需预设数据集。
- 在GSMHard和MATH数据集上达到当前最优查询特定提示效果。
- 适合希望零训练更新大模型但提升推理的开发者使用。
提示优化可在不更新目标模型参数的情况下提升大语言模型的推理能力。现有方法主要分为两类：一类采用基于启发式的‘逐步思考’策略，利用文本反馈从通用大模型中提取改进提示；另一类则通过数值奖励训练专用提示模型，以生成最优提示。本文提出文本奖励提示框架（TRPrompt），将文本反馈直接融入提示模型的训练过程。该框架无需预先收集数据集，可基于生成提示的反馈持续迭代优化。结合大模型对‘好提示’内在理解的能力，文本奖励提供的高分辨率信号使我们训练出在挑战性数学数据集GSMHard和MATH上表现最优的查询特定提示。
原文摘要 · Abstract (English)
Prompt optimization improves the reasoning abilities of large language models (LLMs) without requiring parameter updates to the target model. Following heuristic-based "Think step by step" approaches, the field has evolved in two main directions: while one group of methods uses textual feedback to elicit improved prompts from general-purpose LLMs in a training-free way, a concurrent line of research relies on numerical rewards to train a special prompt model, tailored for providing optimal prompts to the target model. In this paper, we introduce the Textual Reward Prompt framework (TRPrompt), which unifies these approaches by directly incorporating textual feedback into training of the prompt model. Our framework does not require prior dataset collection and is being iteratively improved with the feedback on the generated prompts. When coupled with the capacity of an LLM to internalize the notion of what a "good" prompt is, the high-resolution signal provided by the textual rewards allows us to train a prompt model yielding state-of-the-art query-specific prompts for the problems from the challenging math datasets GSMHard and MATH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。