arXiv:2603.07961cs.CV2026-03ACL

解决视觉场景图生成中关系稀疏与偏差问题,实现端到端无偏生成。

SGG-R$^{\rm 3}$: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation

  • 通过思维链引导的微调与强化学习分阶段优化生成过程。
  • 在两个基准上达到更优性能,关系覆盖率和召回率显著提升。
  • 适合需要高精度、无偏场景理解的视觉推理任务研究者。

场景图生成(SGG)将视觉场景表示为物体及其关系的图结构。尽管多模态大语言模型(MLLM)推动了端到端SGG的发展,但现有方法受限于缺乏任务特定的结构化推理以及关系分布稀疏、长尾等问题,导致生成的场景图召回率低且存在预测偏差。为此,我们提出SGG-R$^{ m 3}$,一个融合任务特定思维链(CoT)引导微调(SFT)与强化学习(RL)的结构化推理框架,采用组序列策略优化(GSPO),分三阶段实现端到端无偏场景图生成。在SFT阶段,利用MLLM生成并基于嵌入相似性过滤的关系增强策略缓解关系稀疏。随后,设计阶段对齐奖励机制,在强化学习中引入细粒度与粗粒度相结合的双重奖励,通过频率自适应加权缓解长尾问题,并借助语义聚类提升关系覆盖。在两个基准上的实验表明,SGG-R$^{ m 3}$优于现有方法,验证了框架的有效性与泛化能力。

原文摘要 · Abstract (English)

Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R$^{\rm 3}$, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R$^{\rm 3}$ achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.

场景图生成多模态强化学习无偏生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。