arXiv:2509.12784cs.CV2025-09

通过上下文表征学习,提升复杂人-物交互检测精度

Contextualized Representation Learning for Effective Human-Object Interaction Detection

  • 用三元组建模工具功能角色,捕捉依赖工具的交互
  • 在HICO-Det和V-COCO上达到更优性能,尤其在复杂场景
  • 适合关注交互理解与视觉语言对齐的研究者

人-物交互(HOI)检测旨在同时定位人-物对并识别其交互行为。尽管近期两阶段方法取得进展,仍受限于上下文建模不充分。本文提出上下文表征学习方法,融合工具引导推理与上下文提示,结合视觉线索以更好捕捉复杂交互。将传统人-物对扩展至包含工具等辅助实体的多变量关系,通过三元组<人, 工具, 物>显式建模工具的功能角色(可操作性),使模型能识别如'填充'这类依赖工具的交互。此外,可学习提示融入实例类别,并通过注意力机制与上下文视觉特征融合,在全局与局部层面实现语言与图像内容对齐。这些上下文表征为复杂、依赖上下文的交互提供更可靠的推理线索。在HICO-Det和V-COCO数据集上,本方法在多数场景表现更优。源代码已开源:https://github.com/lzzhhh1019/CRL。

原文摘要 · Abstract (English)

Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions. While recent two-stage approaches have made significant progress, they still face challenges due to incomplete context modeling. In this work, we introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts with visual cues to better capture complex interactions. We enhance the conventional HOI detection framework by expanding it beyond simple human-object pairs to include multivariate relationships involving auxiliary entities like tools. Specifically, we explicitly model the functional role (affordance) of these auxiliary objects through triplet structures <human, tool, object>. This enables our model to identify tool-dependent interactions such as 'filling'. Furthermore, the learnable prompt is enriched with instance categories and subsequently integrated with contextual visual features using an attention mechanism. This process aligns language with image content at both global and regional levels. These contextualized representations equip the model with enriched relational cues for more reliable reasoning over complex, context-dependent interactions. Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios. The source code is available at https://github.com/lzzhhh1019/CRL.

人-物交互上下文建模视觉语言对齐三元组推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。