让提示词优化自动进化,用代码分析错误并自我修正。
SPEAR: Code-Augmented Agentic Prompt Optimization

- 用可执行的Python代码自主分析评估数据中的错误模式
- 在工业级评测中多项指标超越现有方法,最高提升0.496的κ值
- 适合需要高精度提示优化的工业场景与复杂任务研究者
自动提示工程(APE)通过重写提示词提升下游任务表现,但现有方法将优化器视为固定流程。本文将代码即动作范式引入APE,提出SPEAR(带主动回滚的沙盒提示工程师),一个具备四个工具(评估、Python、设置提示、完成)的自由形式智能体,能自主决定何时使用何种工具。核心是Python沙盒:优化器可编写并执行任意Python代码分析当前评估数据框,自行生成结构化错误分析(混淆矩阵、错误聚类、分组指标)。两个约束机制使长周期智能体保持单调提升:指标下降时自动回滚,可选的最低指标保障。在三个工业级大模型作为评判者的测试集(13项任务,涵盖招聘初筛、对话记忆、查询优化系统)及七项BBH任务和GSM8K上进行评估。SPEAR在所有工业任务主指标上胜出(工具选择κ值0.857对0.359;过滤相关性F1-macro 0.815对0.763;最难提取维度κ 0.254对0.218)。在BBH-7上平均准确率达0.938,远超GEPA的0.628和TextGrad的0.484。消融实验表明,移除Python工具后,5分类工具选择任务κ值下降约0.79,最难提取维度下降约0.35,其不可替代性在于能可靠聚合类别间混淆信息,这是长上下文大模型无法从原始评估数据框中有效提取的。
原文摘要 · Abstract (English)
Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed pipeline. We port the code-as-action paradigm of CodeAct (Wang et al., 2024a) to APE and propose SPEAR (Sandboxed Prompt Engineer with Active Roll-back), a free-form agentic optimizer with four tools -- evaluate, python, set_prompt, finish -- that decides autonomously how and when to use them. The distinctive tool is the Python sandbox: the optimizer writes and executes arbitrary Python on the current evaluation DataFrame, performing structural error analysis (confusion matrices, error clustering, per group metrics) the agent itself authors. Two guardrails turn the long-horizon agent into a monotone-improving optimizer: auto-rollback on metric regression, and an optional guard metric floor. We evaluate on three industrial LLM-as-judge suites (13 judge tasks across recruiter-intake, conversational-memory, and query-refinement systems) plus seven BBH tasks and GSM8K. SPEAR wins every industrial task on the primary metric ($κ$ 0.857 vs 0.359 on tool-selection; F1-macro 0.815 vs 0.763 on filter-relevance; $κ$ 0.254 vs 0.218 on the hardest extraction dimension). On BBH-7 SPEAR averages 0.938 accuracy vs GEPA 0.628 and TextGrad 0.484. Ablations show the Python tool is the largest single lever on complex judge tasks ($Δ\approx +0.79κ$ on the 5-class tool-selection judge, $Δ\approx +0.35κ$ on the hardest extraction dimension when removed); its irreplaceable contribution is class-pair confusion aggregation that a long-context LLM cannot extract reliably from the raw eval DataFrame.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。