通过分析失败模式分布,让大模型工作流自动优化修复缺陷。
Failure-Driven Workflow Refinement
- 将失败视为分布而非简单成败,构建失败签名空间
- 在数学、代码和问答任务上以更低成本提升鲁棒性
- 适合需要高可靠性推理系统的开发者使用
优化基于大语言模型的工作流通常被当作全局搜索问题,仅依赖单一指标评估候选流程。这一范式存在关键缺陷:信息坍塌。将复杂的多步执行轨迹压缩为简单的成功/失败信号,使现有方法无法捕捉失败的内在结构,从根本上阻碍对失败分布的建模。本文将其重构为分布问题,提出新范式:不最大化标量得分,而是直接最小化工作流的期望失败质量,即在高维失败签名空间(FSS)上定义的失败概率密度函数的积分。该分布视角使我们从低效的零阶优化转向对失败景观的有原则梯度下降。我们提出CE-Graph框架,通过反例池近似失败分布,识别密集失败区域作为重复故障模式,并通过提议-验证机制实施受操作符约束的图编辑,贪婪地减少失败质量。在数学、代码和问答基准上,我们的方法在显著更低成本下实现更高鲁棒性,表明系统可靠性并非来自避免失败,而是源于系统性学习并重塑其失败分布的几何结构。
原文摘要 · Abstract (English)
Optimizing LLM-based workflows is typically formulated as a global search, where candidate workflows are evaluated based on a scalar metric. This paradigm, however, suffers from a critical flaw: information collapse. By reducing rich, multi-step execution traces to simple success/failure signals, existing methods are rendered blind to the underlying structure of failures, fundamentally preventing them from modeling the workflow's failure distribution. We reconceptualize this challenge as a distributional problem. We propose a new paradigm where the optimization goal is not to maximize a scalar score, but to directly minimize a workflow's Expected Failure Mass, i.e., the integral of its failure probability density function defined over a high-dimensional Failure Signature Space (FSS). This distributional lens allows us to move from inefficient, zero-order optimization to a principled, gradient-like descent on the failure landscape itself. We introduce CE-Graph, a framework that operationalizes this paradigm through a novel, failure-driven refinement process. CE-Graph approximates the failure distribution from a pool of counterexamples, identifies its densest regions as recurring failure modes, and applies targeted, operator-constrained graph edits via a Propose-and-Verify mechanism to greedily reduce the failure mass. On math, code, and QA benchmarks, our CE-Graph achieves higher robustness at a significantly lower cost than strong baselines. This suggests that a system's reliability emerges not from avoiding failures, but from systematically learning and reshaping the geometric structure of its failure distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。