构建可验证状态的业务流程测试平台，揭示智能体可靠完成任务的差距
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
- 搭建支持多轮交互与状态追踪的沙盒环境，实现端到端任务验证
- 507个带策略约束的工作流测试中，最强模型仅47.53%能稳定通过
- 适合评估企业级智能体在真实业务场景中的可靠性与鲁棒性
当前智能体评测日益依赖可执行环境，涵盖代码修复、网页导航、应用接口调用等。但完成真正有影响的任务不仅需生成合理响应或有效工具调用，还需在多轮交互中获取缺失信息、遵循领域规则、协调依赖工具，并实现正确且无副作用的持久状态转移。本文提出Thinkingbox，一个支持工具-智能体-用户交互的沙盒系统，提供隔离的MCP兼容工具会话、完整执行轨迹及基于终端后端状态的结果评估。基于此，我们构建了Thinkingbox-bench，包含507个跨零售、酒店、汽车保险、新银行内部IT及咨询类IT/HR支持等场景的策略约束工作流。每个尝试均通过任务特定的可执行检查进行评估，接受有效轨迹，拒绝错误、缺失或多余操作；部分任务还额外校验最终响应的必要属性。在专有与开源模型中，表现最佳的Claude Opus 5在pass@1下得分为66.50%，但在pass^20下仅为47.53%。许多失败案例虽显示干净终止与合法状态变更，表明响应或工具调用层面的信号无法准确反映端到端任务完成度。Thinkingbox-bench揭示了偶尔找到成功路径与可靠完成状态化业务任务之间的巨大差距。我们已开源Thinkingbox与Thinkingbox-bench：https://github.com/microsoft/thinkingbox
原文摘要 · Abstract (English)
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest model Claude Opus 5 achieves 66.50% pass@1, but only 47.53% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。