揭秘企业级智能体可靠性来源：关键在专业模型与路由机制，而非验证循环本身。
Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent
- 通过专业模型与路由机制构建任务分解框架，实现分步纠错与动态调整。
- 在三个基准上提升11~15个百分点，尤其在长链工具调用中表现突出。
- 适用于高可靠性需求的企业级应用，尤其适合复杂多步决策场景。
多步企业级智能体任务常因单次推理缺乏中间校验而失败。本文研究生产系统Leni的架构：通过轻量级任务专用后训练模型驱动验证循环（执行、观察、比较、修正），在三个公开基准上评估其性能。在SpreadsheetBench Verified（静默计算错误）上，系统准确率从80.25%提升至91.25%（n=400，p<0.001）；在BullshitBench v2（前提虚构）上提升7~10个百分点（98% vs 91%，n=100）；在GAIA验证集上，通过最佳选择法达到83.0%（pass@1为75.2%，n=165）。核心发现：性能提升主要来自支架结构、路由策略和专用模型，而非验证步骤本身；孤立验证仅贡献+1.5点，但集中在高分段，能挽救原本失败的任务。端到端监控得到混淆矩阵（捕获率约0.20，修复率0.75，无误报回归），支持可靠性递增模型。专家级控制实验显示，前提有效时零过拒。替换验证器为基线模型后，大部分救援失效。
原文摘要 · Abstract (English)
Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it. We study one production system (Leni) whose architecture installs such checkpoints: verification loops (execute, observe, compare, correct) staffed by lightweight task-specialized post-trained models. We evaluate the unmodified production configuration on three public benchmarks stressing distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The full system improves over its frontier base model by +11.0 percentage points on SpreadsheetBench (91.25% vs 80.25%, n=400, p<0.001), +7 to +10 percentage points on BullshitBench (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, n=165; 83.0% best-of-k). Our central contribution is a decomposition of that uplift: most of it comes from scaffolding, routing, and specialist models rather than from the verification step itself, whose isolated contribution is small (+1.5 points) but concentrated at the top of the score distribution, where it converts otherwise-failing tasks. We instrument the loop end-to-end, yielding an empirical verifier confusion matrix (catch rate about 0.20, fix rate 0.75, no false-alarm regressions) that grounds a compounding-reliability model. Specialist-swap ablations suggest that the loop's value depends on who observes it: replacing the small trained verifier with the generating frontier model eliminates most rescues. A valid-premise control shows zero over-rejections in 100 expert-level questions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。