通过关注物体交互关系,提升开放词汇场景图生成的准确性。
Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation
- 以交互为中心设计知识注入与迁移机制,增强模型对物体关系的理解。
- 在三个基准上达到领先性能,显著减少误判和模糊匹配问题。
- 适合需要精准识别复杂场景关系的研究与应用,如自动驾驶、机器人视觉。
开放词汇场景图生成(OVSGG)通过利用预训练大模型的知识,扩展传统方法对未见物体和关系的识别能力。现有方法采用两阶段流程:首先在大规模数据上预训练以注入知识;其次在有标注场景图上微调以转移知识。然而,由于缺乏显式的交互建模,这些方法难以区分同类别中存在交互与无交互的实例,导致知识注入阶段产生噪声伪监督信号,知识迁移阶段引发查询匹配模糊。为此,本文提出一种交互驱动的端到端框架ACC,实现交互中心化的知识注入与迁移。在知识注入阶段,使用双向交互提示生成鲁棒的伪监督信号;在知识迁移阶段,先通过交互引导的查询选择优先配对交互对象,降低非交互对象干扰;再引入一致交互知识蒸馏,将关系前景推向背景之外,同时保留通用知识。在三个基准上的实验表明,ACC取得当前最优性能,验证了交互中心范式在真实场景中的潜力。
原文摘要 · Abstract (English)
Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt a two-stage pipeline: 1) \textit{Infusing knowledge} into large-scale models via pre-training on large datasets; 2) \textit{Transferring knowledge} from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category. This limitation induces critical issues in both stages of OVSGG: it generates noisy pseudo-supervision from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer. To this end, in this paper, we propose an inter\textbf{AC}tion-\textbf{C}entric end-to-end OVSGG framework (\textbf{ACC}) in an interaction-driven paradigm to minimize these mismatches. For \textit{interaction-centric knowledge infusion}, ACC employs a bidirectional interaction prompt for robust pseudo-supervision generation to enhance the model's interaction knowledge. For \textit{interaction-centric knowledge transfer}, ACC first adopts interaction-guided query selection that prioritizes pairing interacting objects to reduce interference from non-interacting ones. Then, it integrates interaction-consistent knowledge distillation to bolster robustness by pushing relational foreground away from the background while retaining general knowledge. Extensive experimental results on three benchmarks show that ACC achieves state-of-the-art performance, demonstrating the potential of interaction-centric paradigms for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。