构建首个面向长时程实体交互的闭环推理基准,评测智能体对物理界面的持续操作与纠错能力。
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
- 设计包含1170段视频的闭环交互数据集,标注指令、动作、状态变化与恢复行为
- 发现主流多模态模型在细粒度时空感知与结果验证上仍存显著缺陷
- 适合评估具身智能体在真实场景中持续交互与错误修复能力的研究者使用
实体控制接口(TCIs),如电器面板、遥控器、电梯按钮和嵌入式图形界面,是日常人造环境的基础组成部分。与这些接口交互要求智能体不仅将语言与视觉观察对齐,还需执行动作、追踪随时间演变的状态变化,并验证目标结果是否达成。然而现有基准大多仅评估开环感知或单步动作执行,未能捕捉这一持续的交互-反馈-修正循环。我们提出SWITCH,一个面向真实第一人称环境中TCI闭环交互推理的基准。SWITCH包含1,170段跨多种功能类别的时序交互视频,提供结构化标注:指令、动作、状态转移、结果及恢复行为。为检验生成式世界建模能力,还通过大模型评分与人工评估,测试视频生成模型在以交互为中心任务上的表现。对前沿专有与开源多模态模型的实验揭示其在细粒度视觉-时序感知、结果验证与错误恢复方面仍存在持续弱点,凸显SWITCH作为闭环具身智能测试床的价值。
原文摘要 · Abstract (English)
Tangible control interfaces (TCIs), such as appliance panels, remotes, elevators, and embedded GUIs, are a fundamental component of everyday human-built environments. Interacting with these interfaces requires agents not only to ground language in visual observations,but also to execute actions, track temporally evolving state changes, and verify whether intended outcomes have been achieved. However, existing benchmarks predominantly evaluate open-loop perception or single-step action execution, failing to capture this continuous cycle of interaction, feedback, and correction. We introduce SWITCH, a benchmark for closed-loop interactive reasoning with TCIs in realistic egocentric environments1. SWITCH comprises 1,170 temporally interactive videos across diverse functional categories, providing structured annotations of instructions, actions, state transitions, outcomes, and recovery behaviors over time. To probe generative world modeling, SWITCH also evaluates video generation models on interaction-centered tasks using both LLM-as-judge and human evaluation2.Experiments with frontier proprietary and opensource multimodal models reveal persistent weaknesses in fine-grained visual-temporal perception, outcome verification, and error recovery, highlighting SWITCH as a testbed for closed-loop embodied intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。