通过分层提示增强图文对齐，提升开放词汇场景图生成效果
Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
- 设计分层提示框架，融合主客体与区域关系信息
- 在Visual Genome和Open Images v6上达到最新最佳性能
- 适合需要细粒度视觉关系理解的研究者
开放词汇场景图生成（OV-SGG）通过将视觉关系表征与开放词汇文本表征对齐，突破了封闭集假设的限制，能够识别新颖的视觉关系，适用于真实世界中多样化的场景。然而，现有方法受限于固定文本表征，导致图文对齐的多样性与准确性不足。为此，我们提出关系感知分层提示（RAHP）框架，通过整合主-客体及区域特定关系信息，增强文本表征。该方法利用实体聚类处理关系三元组类别复杂性，有效融合主-客体信息；同时借助大语言模型（LLM）生成细致的区域感知提示，捕捉细微视觉交互，提升视觉与文本模态间的对齐。此外，RAHP在视觉语言模型（VLM）中引入动态选择机制，根据视觉内容自适应选取相关提示，减少无关提示带来的噪声。在Visual Genome和Open Images v6数据集上的大量实验表明，本框架持续取得最优表现，验证了其在开放词汇场景图生成中的有效性。代码已公开：https://github.com/Leon022/RAHP
原文摘要 · Abstract (English)
Open-vocabulary Scene Graph Generation (OV-SGG) overcomes the limitations of the closed-set assumption by aligning visual relationship representations with open-vocabulary textual representations. This enables the identification of novel visual relationships, making it applicable to real-world scenarios with diverse relationships. However, existing OV-SGG methods are constrained by fixed text representations, limiting diversity and accuracy in image-text alignment. To address these challenges, we propose the Relation-Aware Hierarchical Prompting (RAHP) framework, which enhances text representation by integrating subject-object and region-specific relation information. Our approach utilizes entity clustering to address the complexity of relation triplet categories, enabling the effective integration of subject-object information. Additionally, we utilize a large language model (LLM) to generate detailed region-aware prompts, capturing fine-grained visual interactions and improving alignment between visual and textual modalities. RAHP also introduces a dynamic selection mechanism within Vision-Language Models (VLMs), which adaptively selects relevant text prompts based on the visual content, reducing noise from irrelevant prompts. Extensive experiments on the Visual Genome and Open Images v6 datasets demonstrate that our framework consistently achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of open-vocabulary scene graph generation. The code is available at: https://github.com/Leon022/RAHP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。