解决视觉场景图生成中关系稀疏与偏差问题,实现端到端无偏生成。
SGG-R$^{\rm 3}$: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation
- 通过思维链引导的微调与强化学习分阶段优化生成过程。
- 在两个基准上达到更优性能,关系覆盖率和召回率显著提升。
- 适合需要高精度、无偏场景理解的视觉推理任务研究者。
场景图生成(SGG)将视觉场景表示为物体及其关系的图结构。尽管多模态大语言模型(MLLM)推动了端到端SGG的发展,但现有方法受限于缺乏任务特定的结构化推理以及关系分布稀疏、长尾等问题,导致生成的场景图召回率低且存在预测偏差。为此,我们提出SGG-R$^{ m 3}$,一个融合任务特定思维链(CoT)引导微调(SFT)与强化学习(RL)的结构化推理框架,采用组序列策略优化(GSPO),分三阶段实现端到端无偏场景图生成。在SFT阶段,利用MLLM生成并基于嵌入相似性过滤的关系增强策略缓解关系稀疏。随后,设计阶段对齐奖励机制,在强化学习中引入细粒度与粗粒度相结合的双重奖励,通过频率自适应加权缓解长尾问题,并借助语义聚类提升关系覆盖。在两个基准上的实验表明,SGG-R$^{ m 3}$优于现有方法,验证了框架的有效性与泛化能力。
原文摘要 · Abstract (English)
Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R$^{\rm 3}$, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R$^{\rm 3}$ achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。