构建首个基于人类行为视频的机器人操作评估基准，测试机器对动作意图的理解能力。
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
- 用真实人类操作视频+语言指令匹配模拟场景，实现行为感知的机器人任务评估
- 3000个长时序任务覆盖4大认知维度，最先进模型在真实机器人上成功率仅14%
- 提出分离式评测协议，可分别检验视觉理解、策略执行与端到端系统性能
机器人与人协作时需理解对方的行为顺序与意图。现有基准仅提供单张图像与指令，无法评估对观察行为的推理能力。本文提出WatchAct，一个基于真实人类动作视频的机器人操作基准。每个实例包含一段真实世界人类操作视频、语言指令，以及对应的模拟环境和可执行的LIBERO任务，支持可扩展且可复现的评估。该基准涵盖3000个长时序实例，分布在14项任务中，覆盖四大认知维度：事件解析（Event Grounding）、程序结构恢复（Procedural Reasoning）、隐含意图推断（Implicit Intent Inference）和场景变化追踪（Episodic Reasoning）。我们进一步提出解耦评估协议，分别衡量（i）视觉-语言模型的视频到计划推理能力，（ii）基于理想计划的策略执行表现，（iii）端到端规划-策略系统的整体任务完成度。在仿真和Franka Research 3机器人上，当前系统仍远未解决该基准。最佳方案Gemini-3.1-Pro结合$π_{0.5}$在仿真中成功率达16.3%，真实机器人上为14.0%。Gemini-3.1-Pro的计划成功率仅为36.8%（人类达97.1%），而$π_{0.5}$在理想计划下任务成功率为21.5%，跨域场景下降至10.6%。数据集与代码已公开于https://baiqi-li.github.io/watchact_page/。
原文摘要 · Abstract (English)
A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that language leaves underspecified, yet today's manipulation benchmarks pair an instruction with a single current image, offering no way to evaluate reasoning over observed human behavior. We introduce WatchAct, a benchmark for robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another agent: parsing events (Event Grounding), recovering procedural structure (Procedural Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~video-to-plan reasoning by vision-language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner--policy pipelines. In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $π_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $π_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios. Dataset and code are available at https://baiqi-li.github.io/watchact_page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。