让关系谓词随场景动态调整语义,解决图像中语义歧义问题。
Learning Context-Conditioned Predicate Semantics via Prototype Feedback

- 通过原型反馈机制,让谓词语义根据上下文动态调整。
- 在VG-150和GQA-200上,检测准确率分别提升+1.4和+2.7。
- 适合做视觉场景图生成、细粒度关系理解的研究者使用。
在场景图生成中,核心挑战是处理多义谓词在不同上下文中语义变化的问题。现有方法通过将谓词分解为多个静态原型或检索语义相似样本应对,但这些策略保持谓词表示不变,无法根据图像特定证据重新组织语义,导致在模糊上下文中产生系统性误判。本文提出AlignG,通过原型反馈学习上下文感知的谓词语义。AlignG从每张图像的关系候选中推断上下文相关的谓词语义,并将调整后的语义反馈回重校准关系表示。学习目标将这一调整锚定于全局语义中心,防止语义漂移,同时在场景提供一致关系线索时允许选择性重组。在VG-150和GQA-200上的实验表明,相比当前最优基线,性能持续提升,SGDet设置下F@100分别提高+1.4和+2.7。进一步的可视化显示,每张图像的原型相似性发生有结构的动态变化,原型根据场景证据选择性合并或分离谓词,体现上下文依赖的语义重组。代码已开源。
原文摘要 · Abstract (English)
In scene graph generation, a central challenge is modeling polysemous predicates whose meanings shift across contexts. Prior approaches address this issue by decomposing predicates into multiple static prototypes or retrieving semantically similar exemplars. However, these strategies keep predicate representations static and cannot reorganize semantics to reflect image-specific evidence, leading to systematic confusions in ambiguous contexts. We propose AlignG, which learns context-conditioned predicate semantics via prototype feedback. AlignG infers context-conditioned predicate semantics from the relation candidates within each image and feeds the adapted semantics back to recalibrate relation representations. The learning objective anchors this adaptation to global semantic centers, preventing semantic drift while still allowing selective reorganization when the scene provides consistent relational cues. Experiments on VG-150 and GQA-200 show consistent improvements over state-of-the-art baselines, with F@100 improvements of +1.4 on VG-150 and +2.7 on GQA-200 under SGDet. We further visualize per-image prototype similarity shifts and observe coherent context-dependent reorganization where prototypes selectively merge or separate predicates according to scene evidence. The code is available at https://github.com/Namgyu97/AlignG-SGG.pytorch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。