分离动作识别与物体识别,让机器人更准确理解视频操作指令
Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands

- 先识别动作,再通过轨迹和模糊度筛选关键物体
- 在新物体上达到0.261的BLEU-4,比基线高143.9%
- 适合需要精准操作指令的机器人任务,如工业装配
将视频示范转化为可执行机器人命令仍具挑战,因现有方法常无法识别动作中涉及的功能性物体,导致生成的语言通顺但操作模糊。本文提出一种解耦的动作-物体中心视频理解框架,将动作识别与物体识别分离,生成精确、无语法约束的操作指令。方法结合时序移位模块(TSM)实现高效时空动作分类,并引入新颖的物体选择算法,通过轨迹角色分类、模糊检测与重叠最小化识别任务相关物体;选定物体由视觉-语言模型(VLM)处理,实现鲁棒类别识别与零样本泛化。在改进版Something-Something V2数据集上,该方法动作分类准确率达86.79%,标准物体上BLEU-4为0.337,新物体上为0.261,分别较最强任务专用基线提升80.2%和143.9%。在METEOR与CIDEr指标上,新物体上的提升分别达157.9%和171.7%。所有语义指标均优于任务专用方法,且在保持模块化物体中心设计的同时,性能媲美甚至超越大型通用VLM。
原文摘要 · Abstract (English)
Translating video demonstrations into executable robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel \textbf{Object Selection} algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by Vision-Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2 dataset, our method achieves 86.79\% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2\% and 143.9\%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9\% and 171.7\% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。