arXiv:2605.14102cs.AI2026-05

复杂工具调度反而降低智能体表现,揭示可靠评估的底层约束

ChromaFlow: A Negative Ablation Study of Orchestration Overhead in Tool-Augmented Agent Evaluation

  • 以规划驱动执行+遥测反馈构建评估框架
  • 更激进调度使准确率从54.72%降至50.94%,故障增多
  • 强调确定性提取与健康检查是可靠评估的必备条件

自主语言模型智能体日益融合规划、工具调用、文档处理、浏览、代码执行与验证循环。这些能力虽提升实用性,但也引入仅凭最终准确率无法察觉的运行故障模式。本文提出ChromaFlow,一个基于规划驱动执行、专用工具使用与遥测驱动评估的工具增强型自主推理框架。在GAIA 2023 Level-1验证任务下,冻结全量基准模型取得29/53正确答案(54.72%)。后续扩展调度配置虽增加运行开销,但准确率下降至27/53(50.94%),同时触发更多回溯、超时、工具失败记录、日志调用及成本估算。两次随机20任务烟雾测试分别得到12/20和11/20正确答案,表明微小诊断收益在样本间不稳定。核心结论为负向消融:更激进的编排未提升整体性能且加剧操作噪声。后期严格提供者全级别诊断达30/53(56.60%),但在更高令牌日志成本下实现。报告主张,有限规划升级、确定性提取、证据对齐、提供者健康门控与显式运行门控应视为可靠自治智能体评估的一阶要求。

原文摘要 · Abstract (English)

Autonomous language-model agents increasingly combine planning, tool use, document processing, browsing, code execution, and verification loops. These capabilities make agent systems more useful, but they also introduce operational failure modes that are not visible from final accuracy alone. This report presents ChromaFlow, a tool-augmented autonomous reasoning framework built around planner-directed execution, specialized tool use, and telemetry-driven evaluation. We analyze ChromaFlow on GAIA 2023 Level-1 validation tasks under clean evaluation constraints. A frozen full Level-1 baseline achieved 29/53 correct answers, or 54.72%. A later recovery configuration with expanded orchestration achieved 27/53 correct answers, or 50.94%, while increasing tracebacks, timeout events, tool-failure mentions, token-log calls, and campaign-log cost estimates. Two randomized 20-task smoke evaluations produced 12/20 and 11/20 correct answers, showing that small diagnostic gains can be unstable across samples. The central result is therefore a negative ablation: more aggressive orchestration did not improve full-set performance and increased operational noise. A later strict-provider full-Level-1 diagnostic reached 30/53, or 56.60%, under explicit integrity controls, but at substantially higher token-log cost. The report argues that bounded planner escalation, deterministic extraction, evidence reconciliation, provider-health gates, and explicit run gates should be treated as first-order requirements for reliable autonomous agent evaluation.

智能体评估工具调度可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。