评测大模型智能体在复杂任务中的真实短板,发现其工具依赖与决策稳定性问题。
$C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking
- 设计三类挑战:工具关系、隐藏信息、动态路径,模拟真实交互复杂性
- 49个主流智能体测试显示,工具依赖与长上下文处理能力普遍不足
- 提供可复现的评估方法和细粒度指标,适合研究智能体鲁棒性与可解释性
基于大语言模型的智能体通过调用工具改变环境,推动AI与物理世界交互方式革新。与仅依赖历史对话的传统NLP任务不同,这类智能体需综合考虑工具间关系、环境反馈及先前决策等复杂因素。现有研究多以多轮对话评估智能体,却忽略了这些关键因素对其行为的影响。为填补空白,我们提出开源高质量基准C³-Bench,融合攻击思想并采用单变量分析定位影响智能体鲁棒性的核心要素。具体设计三项挑战:处理复杂工具关系、应对关键隐藏信息、管理动态决策路径。配套引入细粒度评估指标、创新数据采集算法和可复现评估流程。在49个主流智能体(涵盖通用快速思维、慢速思维及领域专用模型)上开展广泛实验,发现其在工具依赖、长上下文依赖及频繁策略切换方面存在显著缺陷。C³-Bench旨在通过挑战暴露模型漏洞,推动智能体性能可解释性研究。基准已公开于https://github.com/TencentHunyuan/C3-Benchmark。
原文摘要 · Abstract (English)
Agents based on large language models leverage tools to modify environments, revolutionizing how AI interacts with the physical world. Unlike traditional NLP tasks that rely solely on historical dialogue for responses, these agents must consider more complex factors, such as inter-tool relationships, environmental feedback and previous decisions, when making choices. Current research typically evaluates agents via multi-turn dialogues. However, it overlooks the influence of these critical factors on agent behavior. To bridge this gap, we present an open-source and high-quality benchmark $C^3$-Bench. This benchmark integrates attack concepts and applies univariate analysis to pinpoint key elements affecting agent robustness. In concrete, we design three challenges: navigate complex tool relationships, handle critical hidden information and manage dynamic decision paths. Complementing these challenges, we introduce fine-grained metrics, innovative data collection algorithms and reproducible evaluation methods. Extensive experiments are conducted on 49 mainstream agents, encompassing general fast-thinking, slow-thinking and domain-specific models. We observe that agents have significant shortcomings in handling tool dependencies, long context information dependencies and frequent policy-type switching. In essence, $C^3$-Bench aims to expose model vulnerabilities through these challenges and drive research into the interpretability of agent performance. The benchmark is publicly available at https://github.com/TencentHunyuan/C3-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。