通过分解组装动作提升机器人理解复杂操作能力
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
- 将动作拆解为动词、物体、工具三要素，用模板问答微调视觉语言模型
- 在有限数据下实现多任务学习，准确率显著高于现有基线方法
- 提供可解释的元素级预测，适合人机协作与智能装配系统
组装动作理解是实现人机协同装配的关键，但因动作细微且手物交互精细而极具挑战。本文提出组合上下文微调（CCFT）方法，将组装动作分解为语义元素（动词、物体、工具），并通过模板化问题-回答对微调视觉语言模型以识别各元素，确保输出近似确定性。为在数据有限条件下实现高效多任务学习，提出分层分区交替训练（LP-AT）策略：通过特定元素的低秩适配器分配不同模型层，并交替更新适配器权重，降低任务间干扰，同时支持每适配器独立超参数优化。此外，基于现有装配视频数据集构建了HA-ViD-VQA和IKEA-ASM-VQA两个新数据集。大量实验表明，本方法在多个数据集上持续优于强基线，在保持高精度的同时提供可解释的元素级预测，适用于多样化下游应用。
原文摘要 · Abstract (English)
Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。