把提示词优化当成搜索问题,用算法自动找更好的提示。
Prompt Optimization as a State-Space Search Problem
- 将提示词空间建模为图,通过剪枝搜索高效寻找最优提示。
- 浅层搜索(宽2深2)在5个任务上提升开发集准确率,最高达0.80。
- 简洁化操作最有效,冗余添加从不被选中,适合调试提示词的工程师。
语言模型对提示词微小变化极为敏感,易导致性能崩溃。现有工具如DSpy通过示例驱动优化提示词。本文提出将提示词优化视为经典的状态空间搜索问题:将提示词空间建模为图,节点代表提示状态,边对应刻意变换(如缩短、添加示例、重排内容)。采用束搜索和随机游走算法系统探索该空间,在开发集上评估候选提示并剪枝低效分支。在五个自然语言任务(情感分类、问答、摘要、推理、自然语言蕴含)上,即使浅层搜索配置(束宽=2,深度=2),开发集表现也优于初始提示。例如,推理任务开发集准确率从0.40提升至0.80,但测试集提升较温和(0.20→0.50),表明对开发集启发式存在过拟合。成功优化路径分析显示,使提示更简洁的操作出现频率最高,而冗余扩展从未被选中。结果验证了提示优化作为搜索问题的可行性,并指出更大计算资源与更好评估指标下,可实现更泛化的鲁棒提示。代码已开源。
原文摘要 · Abstract (English)
Language Models are extremely susceptible to performance collapse with even small changes to input prompt strings. Libraries such as DSpy (from Stanford NLP) avoid this problem through demonstration-based prompt optimisation. Inspired by this, I propose an alternative approach that treats prompt optimisation as a classical state-space search problem. I model the prompt space as a graph where nodes represent prompt states and edges correspond to deliberate transformations such as shortening, adding examples, or re- ordering content. Using beam search and random walk algorithms, I systematically explore this space, evaluating candidates on development sets and pruning unpromising branches. Across five NLP tasks (sentiment classification, question answering, summarisation, reason- ing, and natural language inference), I find that even shallow search configurations (beam width=2, depth=2) improve upon seed prompts on development sets. For instance, beam search achieves development accuracy gains from 0.40 to 0.80 on reasoning tasks, though test set improvements are more modest (0.20 to 0.50), indicating overfitting to the develop- ment heuristic. Analysis of successful optimisation paths reveals that transformations that make prompts concise appear most frequently, while verbosity operators are never selected. My results validate prompt optimization as a search problem and suggest that with greater computational resources and improved evaluation metrics, deeper exploration could yield more robust prompts that generalize beyond development sets. Code and implementation are available at [https://github.com/MaanasTaneja/PromptOptimiser].
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。