arXiv:2605.12947stat.MLcs.AI2026-05

为黑箱生成-验证系统设计始终有效的释放机制,避免过早输出错误结果。

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems

  • 构建高分失败样本池作为保守参照,校准部署时评估分数。
  • 用e过程累积证据,在任意停止时机下保证释放可靠性。
  • 实测在编程任务中减少误释放,同时仍能识别可解任务。

基于大模型的AI工作流常通过迭代生成-评估-修正循环产出结果。每次迭代可能提升候选结果,但也带来释放决策难题:何时停止并输出当前结果?这引发统计挑战,因部署时评估分数是自适应生成并持续监控的,而通常用于校准的似然模型或交换性假设不可用。本文提出一种适用于现有生成-评估流水线的始终有效释放封装器。该封装器构建一个高分失败样本的硬负样本池,将部署时评估分数与此池校准,并以e过程累积所得证据。该设计分离了两个角色:参考池将黑箱评分转化为保守证据,而e过程确保在可选停止条件下依然有效。理论上,我们证明保守参考池可实现对无法完成任务(即工作流无法生成可靠解)的释放概率的有限样本控制;同时刻画了该保守规则仍能在可行任务上实现非平凡释放的条件。在MBPP+编程代理的案例研究中,该封装器相较于基线停止规则显著降低过早错误释放,同时仍能对工作流反复积累中等支持证据的任务进行释放。

原文摘要 · Abstract (English)

LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but it also creates a release decision: when to stop and output the current result? This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored, yet the likelihood models or exchangeability assumptions typically used for calibration are unavailable. We propose an always-valid release wrapper for existing generator-evaluator pipelines. The wrapper builds a hard-negative reference pool of high-scoring failures, calibrates deployment-time evaluator scores against this pool, and accumulates the resulting evidence with an e-process. This separates two roles: the reference pool turns black-box scores into conservative evidence, while the e-process provides validity under optional stopping. In theory, we show that a conservative reference pool yields finite-sample control of the probability of releasing on infeasible tasks, that is, tasks for which the given workflow is not capable of producing a reliable solution. We also characterize conditions under which the same conservative rule still achieves nontrivial release on feasible tasks. In an MBPP+ coding-agent case study, the wrapper reduces premature incorrect release relative to baseline stopping rules while still releasing on tasks for which the workflow repeatedly accumulates moderate supporting evidence.

AI工作流释放策略统计验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。