自动生成任务专属提示指南,让大模型回答更精准。
Automatically Evolving Prompt Guidelines for Task-Specific Optimization
- 用优化算法自动生成适配具体任务的提示编写规范。
- 在数学、医疗和编程任务中提升性能15.5%至81.7%。
- 适合需要高精度输出的场景,如专业问答与代码生成。
为使大语言模型可靠回应用户查询,用户需明确说明要求、上下文和约束。然而实际中,用户查询常不完整,迫使模型推断未明说假设,可能导致与真实意图不符。现有提示工程指南多为通用且任务无关,且手工制定、缺乏系统性。为此,本文研究提示指南优化:自动生成针对特定任务和模型的指导原则,以帮助撰写更清晰的提示。核心观察是已完成的任务示例(参考答案)往往隐含完成不完整查询所需的信息,包括行为约束、上下文假设和评估标准。为此提出AGOPS方法,通过提示生成器LLM、求解器LLM与提示演化相结合的优化机制,最大化在一组示例(用户查询与参考答案)上的下游表现。推理时,这些指南可帮助用户写出更规范提示,显著提升模型效果。实验表明,在数学推理、医疗问答和编码任务中,提示不完整导致性能下降高达95.3%,且现有优化技术几乎无法恢复;而遵循AGOPS指南的用户可平均提升15.5%至81.7%的性能,跨基准一致有效。
原文摘要 · Abstract (English)
For Large Language Models to reliably answer user queries, users must clearly specify requirements, context, and constraints. In practice, however, user queries are often underspecified, forcing models to infer unstated assumptions that may misalign with the actual user intent. Existing prompt engineering guidelines aim to mitigate this issue, they are typically generic and task-agnostic, limiting their practical utility. Additionally, existing guidelines are formed manually and in a non-systematic way. To this end, we study prompt guideline optimization: the problem of automatically generating task-specific guidelines that help write better-specified prompts for a given task and model. Our key observation is that existing (completed) task examples (aka reference answers) often implicitly encode the missing information required to complete underspecified queries, including behavioral constraints, contextual assumptions, and evaluation criteria. We therefore propose AGOPS, an automatic approach that evolves task-specific guidelines via an optimization scheme that involves a prompt LLM writer, a solver LLM and prompt evolution, which maximize downstream effectiveness on a set of examples (user queries with reference answers). At inference time, our guidelines help users write well-specified prompts, boosting the effectiveness of LLMs. We show across mathematical reasoning, medical question answering, and coding tasks, that prompt underspecification leads to major drops (up to 95.3%) in downstream task performance (compared to well-specified prompts) and, perhaps more importantly, that this drop can hardly be recovered by existing prompt optimization techniques. Users following AGOPS guidelines can regain this loss (increasing performance between 15.5 to 81.7% on average) consistently across all benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。