让大模型学会看图中的物体关系，提升视觉推理能力。
Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
- 用场景图结构显式表示图像中物体与关系
- 构建12万条带推理路径的数据，训练模型理解结构化信息
- 适合需要精细视觉推理的多模态任务，如导航、问答
多模态大语言模型虽具备强大感知与推理能力，但多数模型仅关注孤立物体，忽视结构化关系，限制了其在视觉密集型任务中的表现。为此，我们提出场景图思维（SaGe）新范式，通过显式场景图表示实现细粒度、结构化的视觉推理。首先设计自动化数据引擎，将平面图文语料转换为结构化场景图，其中层级实体为节点，多样视觉关系为边。基于此，采样生成12万条高质量训练数据。随后引入两阶段图对齐后训练：监督微调使模型内化结构化推理能力，强化学习微调则以节点为代理奖励，促进高效图探索。该方法在八个多模态基准上显著提升性能，尤其在细粒度感知与推理任务中表现优异。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。