自动优化多步大模型流程,提升准确率14.1个百分点。
FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelines
- 通过诊断中间步骤故障,动态调整提示词或流程结构
- 在18组对比中15胜,安全任务最高提效7.1个百分点
- 适合需要可靠多步推理的通用与安全场景
多步大模型流水线因检索、推理和格式化步骤间交互而失败,仅优化提示词可能遗漏链路瓶颈。本文提出完全自动化提示优化框架FAPO,使Claude Code可在标准化代码库内优化大模型流水线。FAPO评估流水线,检查中间步骤,诊断失败,提出针对性修改,并反复验证变体以优化得分函数。先尝试提示词修改,仅当提示优化不足时,在可接受范围内调整链路结构,由归因分析识别结构性瓶颈。在六个基准测试和三个任务模型上,FAPO在18组对比中有15组优于基线GEPA;其中11组结果均值差异显著,平均提升14.1个百分点。在六组HoVer与IFBench测试中,提示优先搜索升级为结构变更时,FAPO全胜,平均提升33.8个百分点。在安全任务上亦有显著提升:在CTIBench-RCM(CVE到CWE映射)任务中,提示词优化使GPT-5准确率提升+4.0个百分点,Foundation-Sec-8B-Instruct提升+7.1个百分点,Foundation-Sec-8B-Reasoning提升+2.0个百分点。这些结果表明,FAPO是通用与安全任务的顶尖流水线优化技术。
原文摘要 · Abstract (English)
Multi-step LLM pipelines fail through interactions among retrieval, reasoning, and formatting steps, so prompt-only optimization can miss bottlenecks in the chain. We present Fully Automated Prompt Optimization (FAPO), a framework that lets Claude Code optimize an LLM pipeline inside a standardized codebase. FAPO evaluates a pipeline, inspects intermediate steps, diagnoses failures, proposes scoped changes, and validates variants repeatedly to optimize against a score function. It first tries prompt edits and, only when prompt optimization appears insufficient, changes chain structure within the permitted scope when attribution identifies a structural bottleneck. Across six benchmarks and three task models, FAPO beats the baseline GEPA in 15 of 18 model-benchmark comparisons. In 11 model-benchmark comparisons, FAPO wins with non-overlapping mean $\pm$ trial-standard-deviation ranges, and the mean FAPO-GEPA gain is +14.1 pp. In the six HoVer and IFBench comparisons where prompt-first search escalated to structural changes, FAPO wins all six with a mean gain of +33.8 pp. FAPO also improves performance on security tasks: on CTIBench-RCM, a security CVE-to-CWE task, prompt-only FAPO lifts test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning. These results position FAPO as a state-of-the-art pipeline optimization technique for both general-purpose and security-focused tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。