首个面向真实世界任务的主动智能体通用评测基准,可精准评估五大核心能力。
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

- 按五项基础能力设计400个双语现实任务,覆盖技能使用、探索、长上下文推理等
- 在实时Docker容器中逐步评估,通过闭环机制模拟真实多轮人机反馈
- 支持模型与框架分离评测,助力理解性能瓶颈来源,适合智能体研究者使用
大语言模型和多模态大语言模型的快速发展推动了能够在真实环境中操作日常工具的主动智能体的出现。然而,现有评测基准难以有效评估此类智能体,因其常依赖沙盒环境和单轮评估范式,且任务分类混杂多种模型能力,难以定位失败根源。为此,我们提出UniClawBench,首个以能力为导向的基准,用于评估动态真实世界中的主动智能体。该基准围绕五项基础能力构建:技能使用、探索、长上下文推理、多模态理解与跨平台协同。基于此设计了400个双语真实任务。不同于依赖静态预录答案的基准,本基准在实时Docker容器中进行细粒度步骤评估。此外,设计了包含执行代理、隐藏监督代理与用户代理的闭环评估策略,模拟真实多轮人类反馈而不泄露评分标准。为分离基础模型能力与框架设计影响,我们在多个智能体框架下评测前沿模型。综合对比显示,基础模型能力与智能体框架设计共同决定真实环境表现。为促进后续研究,基准代码已公开于https://github.com/HKU-MMLab/UniClawBench。
原文摘要 · Abstract (English)
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。