评测大模型在多任务中异步调用工具的能力,发现延迟响应会严重拖累性能。
AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios

- 构建多任务异步调用基准AsyncTool,模拟真实延迟反馈
- 实测显示延迟导致模型性能明显下降,需更好任务协调能力
- 适合关注大模型多任务效率与时间推理的研究者
基于大语言模型(LLM)的智能体在使用外部工具解决复杂任务方面展现出强大能力。然而,现有评估通常忽略工具使用的时序特性,尤其未考虑工具响应延迟的影响,且多限于单任务场景。在真实应用中,多个任务常需并发执行,整体效率取决于智能体能否在等待工具响应时有效利用空闲时间。我们将此能力称为异步工具调用。为此,我们提出AsyncTool,一个用于评估LLM智能体在存在延迟反馈的交互式多任务工具使用环境中的基准。AsyncTool同时呈现多个异构任务,并在执行过程中模拟真实的工具响应延迟。通过混合数据演化策略,我们构建了一个涵盖多种场景和工具使用模式的多样化异步多任务数据集。我们在步骤、子任务和任务三个层级评估模型,并引入面向效率的指标来衡量任务协调与完成效率。大量实验表明,延迟反馈对当前智能体构成显著挑战,导致性能明显下降。能够更好进行任务切换、依赖追踪和状态维护的模型在AsyncTool上表现更优。我们的分析揭示了当前工具使用智能体的关键失败模式,并为设计具备更强时序推理与协调能力的未来系统提供了实用洞见。
原文摘要 · Abstract (English)
Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks. However, existing evaluations often overlook the temporal dimension of tool use, especially the impact of tool response latency, and are usually limited to single-task settings. In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency depends on whether an agent can use idle time while waiting for tool responses. We refer to this capability as asynchronous tool calling. To evaluate it, we propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback. AsyncTool presents multiple heterogeneous tasks simultaneously and simulates realistic tool response latency during execution. Using a hybrid data evolution strategy, we construct a diverse asynchronous multitasking dataset that covers multiple scenarios and tool-use patterns. We evaluate models at the step, sub-task, and task levels, and introduce efficiency-oriented metrics to measure task coordination and completion efficiency. Extensive experiments show that delayed tool feedback poses substantial challenges to current agents and leads to clear performance degradation. Models that better coordinate task switching, dependency tracking, and state maintenance achieve stronger performance on AsyncTool. Our analysis identifies key failure modes of current tool-using agents and provides practical insights for designing future systems with stronger temporal reasoning and coordination capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。