构建真实复杂的助理任务评估基准，兼顾任务分布与执行语义的双重真实性
LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks
- 基于三轴复杂度框架设计任务，模拟跨服务依赖等真实挑战
- 提供可复现的全栈模拟应用，保留状态与执行流程
- 支持细粒度诊断，适合评估复杂代理任务的鲁棒性
OpenClaw风格的个人助理将大模型代理从孤立工具调用扩展到开放、有状态且个性化的软件环境。评估这类助理本质上是保真度问题：基准必须忠实于真实助理任务的分布，以及任务执行环境的语义。现有基准往往在某一维度上失真——任务分布受易隔离、模拟和验证的限制，忽略跨服务依赖、状态污染、隐含意图和运行时变化等现实难题；环境要么是实时但难以复现，要么是仅保留接口层的简化模拟，缺失会话、产物、状态转移及下游副作用。本文提出LiveClawBench，一个满足双保真要求的基准。它结合三轴复杂度框架进行难度驱动的任务构造，并采用可复现的全栈模拟应用以保留状态化执行语义。该基准包含134个可执行案例，覆盖10个领域、22个模拟服务，支持可控、可扩展、因子级的诊断评估。我们开源了基准资源：(1) 基准代码：https://github.com/Mosi-AI/LiveClawBench；(2) 排名榜：https://mosi-ai.github.io/LiveClawBench/leaderboard/；(3) 任务轨迹数据集：https://huggingface.co/datasets/Mosi-AI/LiveClawbench-trajectories
原文摘要 · Abstract (English)
OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments. Evaluating these assistants is fundamentally a fidelity problem: benchmarks must be faithful both to the distribution of real assistant tasks and to the execution semantics of the environments in which those tasks unfold. Existing benchmarks often lose fidelity in one dimension or the other. Their task distributions are shaped by what is easy to isolate, mock, and verify, underrepresenting real-world difficulties such as cross-service dependency, contaminated state, implicit intent, and runtime change. Their environments are either live but hard to reproduce, or reproducible but reduced to endpoint-level stubs that remove sessions, artifacts, state transitions, and downstream side effects. We introduce LiveClawBench, a benchmark designed around this dual-fidelity requirement. LiveClawBench combines a Triple-Axis Complexity Framework for difficulty-driven task construction with reproducible full-stack mock applications that preserve stateful execution semantics. With 134 executable cases across 10 domains with 22 mocked services, LiveClawBench supports controlled, extensible, and factor-level diagnostic evaluation of realistic agentic tasks. We release the benchmark resources: (1) Benchmark: https://github.com/Mosi-AI/LiveClawBench; (2) Leaderboard: https://mosi-ai.github.io/LiveClawBench/leaderboard/; (3) Trajectories: https://huggingface.co/datasets/Mosi-AI/LiveClawbench-trajectories
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。