arXiv:2608.20690cs.CVcs.AI2026-08

让动作描述带上人物身份,实现精准识别的交互动作生成。

Identity-Aware Human-Object Interaction Motion Captioning

论文配图:Identity-Aware Human-Object Interaction Motion Captioning
图 1 · 摘自论文原文
  • 基于多视角视频学习身份与动作关联,支持单视角生成带身份的动作描述。
  • 在BEHAVE和InterCap数据集上达到当前最优效果,准确率显著提升。
  • 适合需要精确追踪人物行为的应用,如智能监控、虚拟助手。

现有行人-物体交互(HOI)动作描述方法通常使用“一个人”或“某人”等泛化术语指代主体,未将描述与具体身份绑定。为此,本文提出身份感知的交互动作描述任务,要求生成的描述同时包含主体身份与交互动作。例如输出“Sub_ID lifts the chair”而非“a person lifts the chair”。基于BEHAVE与InterCap数据集构建了身份感知的HOI动作描述数据集。提出ID-HOINet模型,通过多视角视频学习身份与动作特征,支持单视角生成。其核心包括:多视角身份-动作学习模块(MVIML),用于建模时序阶段与视点间的依赖关系;两阶段重写策略(TSCR),先生成无身份的动作描述,再融合预测身份进行重写。实验表明,该模型在多个指标上达到领先水平。代码将在录用后公开。

原文摘要 · Abstract (English)

Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates "Sub_ID lifts the chair" rather than "A person lifts the chair". For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance.

动作描述身份感知多视角学习交互生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。