用检索增强方法解决场景图生成中的细粒度关系遗漏问题
RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning
- 通过检索记忆库补充缺失的细粒度关系标签
- 在VG和GQA数据集上提升达3.6%和5.9%
- 适合关注长尾分布与语义模糊问题的研究者
场景图生成（SGG）面临两个根本挑战：谓词分布长尾化与谓词间语义模糊。这导致模型偏向头部谓词，忽略细粒度谓词。本文将SGG建模为部分标注的多标签分类问题，提出检索增强的场景图生成框架RA-SGG。该方法识别可能需多标签的实例，从记忆库中检索语义相似样本，补充原始单标签为多标签。基于增强的关系，采用多原型学习训练模型。大量实验表明，RA-SGG在VG上超越现有基线最高3.6%，在GQA上提升5.9%，尤其在F@K指标上表现优异，有效缓解了因谓词长尾分布与语义模糊导致的预测偏差。
原文摘要 · Abstract (English)
Scene Graph Generation (SGG) research has suffered from two fundamental challenges: the long-tailed predicate distribution and semantic ambiguity between predicates. These challenges lead to a bias towards head predicates in SGG models, favoring dominant general predicates while overlooking fine-grained predicates. In this paper, we address the challenges of SGG by framing it as multi-label classification problem with partial annotation, where relevant labels of fine-grained predicates are missing. Under the new frame, we propose Retrieval-Augmented Scene Graph Generation (RA-SGG), which identifies potential instances to be multi-labeled and enriches the single-label with multi-labels that are semantically similar to the original label by retrieving relevant samples from our established memory bank. Based on augmented relations (i.e., discovered multi-labels), we apply multi-prototype learning to train our SGG model. Several comprehensive experiments have demonstrated that RA-SGG outperforms state-of-the-art baselines by up to 3.6% on VG and 5.9% on GQA, particularly in terms of F@K, showing that RA-SGG effectively alleviates the issue of biased prediction caused by the long-tailed distribution and semantic ambiguity of predicates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。