arXiv:2602.20597cs.CV2026-02被引 5

提升第一视角手物交互解析精度,解决模型误判物理合理性问题。

Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing

  • 通过空间接触动态初始化查询,精准定位手与物体的交互
  • 在EgoHOS和mini-HOI4D上达到当前最优性能,泛化能力更强
  • 引入关系约束损失,防止生成不合理的交互预测

第一视角人-环境交互的细粒度理解对下一代具身智能体的发展至关重要。该任务的核心挑战在于准确解析手部与活动物体。尽管基于Transformer的架构在此类任务中展现出巨大潜力,但仍存在三大关键局限:1)现有查询初始化机制主要依赖语义线索或可学习参数,对不同场景下变化的活动物体适应性差;2)以往方法使用像素级语义特征迭代优化查询,可能引入与交互无关的内容;3)现有模型易产生“交互幻觉”,导致物理不一致的预测。为此,我们提出端到端的交互感知Transformer(InterFormer),包含三个核心组件:动态查询生成器(DQG)、双上下文特征选择器(DFS)和条件共现(CoCo)损失。DQG将查询初始化显式地锚定于手物接触的空间动态,实现对不同活动物体的交互感知查询生成。DFS融合粗粒度交互线索与语义特征,抑制非交互噪声,强化交互关系学习。CoCo损失引入手物关系约束,提升预测的物理一致性。模型在EgoHOS和具有挑战性的分布外mini-HOI4D数据集上均取得当前最优表现,验证了其有效性与强泛化能力。代码与模型已公开于https://github.com/yuggiehk/InterFormer。

原文摘要 · Abstract (English)

A fine-grained understanding of egocentric human-environment interactions is crucial for developing next-generation embodied agents. One fundamental challenge in this area involves accurately parsing hands and active objects. While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed: 1) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrating limited adaptability to changing active objects across varying input scenes; 2) previous transformer-based methods utilize pixel-level semantic features to iteratively refine queries during mask generation, which may introduce interaction-irrelevant content into the final embeddings; and 3) prevailing models are susceptible to "interaction illusion", producing physically inconsistent predictions. To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormer), which integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss. The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interaction-aware queries for hands and various active objects. The DFS fuses coarse interactive cues with semantic features, thereby suppressing interaction-irrelevant noise and emphasizing the learning of interactive relationships. The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets, demonstrating its effectiveness and strong generalization ability. Code and models are publicly available at https://github.com/yuggiehk/InterFormer.

第一视角手物交互视觉解析Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。