用语言理解预测手部动作轨迹，让模型能听懂指令并推理出手下一步动向。
HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction
- 融合视觉语言模型与自回归生成，实现自然语言对话下的手部轨迹预测
- 在新基准VHP和RBHP上超越现有方法，显著提升推理能力
- 适合关注人机交互、具身智能与多模态推理的研究者
如何根据自然语言的任务描述，预测场景中人手未来的交互轨迹？本文将经典的手部轨迹预测任务拓展为两个涉及显式或隐式语言查询的新任务。这些任务需要对日常活动有深入理解，并基于当前场景线索进行下一步行为推理。为此，我们构建了两个新基准：基础手部预测（VHP）与基于推理的手部预测（RBHP）。通过结合视觉语言模型（VLM）的高层世界知识与推理能力，以及低层自回归手部轨迹特性，我们提出新型模型HandsOnVLM。该模型可支持自然语言对话，生成文本回复并预测未来手部轨迹。实验表明，HandsOnVLM在所提任务上优于现有专用方法与其他VLM基线，展现出有效利用上下文知识进行低层手部轨迹推理的能力。代码与视频结果详见官网：https://www.chenbao.tech/handsonvlm/
原文摘要 · Abstract (English)
How can we predict future interaction trajectories of human hands in a scene given high-level colloquial task specifications in the form of natural language? In this paper, we extend the classic hand trajectory prediction task to two tasks involving explicit or implicit language queries. Our proposed tasks require extensive understanding of human daily activities and reasoning abilities about what should be happening next given cues from the current scene. We also develop new benchmarks to evaluate the proposed two tasks, Vanilla Hand Prediction (VHP) and Reasoning-Based Hand Prediction (RBHP). We enable solving these tasks by integrating high-level world knowledge and reasoning capabilities of Vision-Language Models (VLMs) with the auto-regressive nature of low-level ego-centric hand trajectories. Our model, HandsOnVLM is a novel VLM that can generate textual responses and produce future hand trajectories through natural-language conversations. Our experiments show that HandsOnVLM outperforms existing task-specific methods and other VLM baselines on proposed tasks, and demonstrates its ability to effectively utilize world knowledge for reasoning about low-level human hand trajectories based on the provided context. Our website contains code and detailed video results https://www.chenbao.tech/handsonvlm/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。