将场景图生成从分类转向渐进式生成,提升关系预测准确率。
Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching

- 用流匹配逐步生成图像条件下的场景图,联合优化物体与关系。
- 在VG和PSG数据集上,谓词准确率平均提升约3个百分点。
- 适合需要高质量视觉关系推理的研究者和开发者。
场景图生成(SGG)通过预测边界框和主语-谓语-宾语三元组,统一了目标定位与视觉关系推理。然而,现有方法大多将SGG视为一次性、确定性的分类任务,而非真正的渐进式生成过程。本文提出FlowSG,将SGG重构为混合离散-连续状态上的连续时间传输问题:从噪声图开始,模型通过约束感知的精化步骤,逐步生成图像条件下的场景图,同时合成节点(物体)与边(谓语)。具体而言,首先利用VQ-VAE将场景图(如连续视觉特征)量化为紧凑且可预测的标记;随后,图Transformer(i)预测条件速度场以传输连续几何(边界框),(ii)更新离散后验概率以优化类别标记(物体特征与谓语标签),通过流条件消息聚合耦合语义与几何。训练结合几何的流匹配损失与标记的离散流目标,实现少步推断,并与标准检测器和分割器即插即用。在VG和PSG数据集上,于封闭与开放词汇协议下进行的大量实验表明,该方法在谓词准确率与图级指标上均持续优于基线,平均比当前最优方法USG-Par提升约3个百分点。
原文摘要 · Abstract (English)
Scene Graph Generation (SGG) unifies object localization and visual relationship reasoning by predicting boxes and subject-predicate-object triples. Yet most pipelines treat SGG as a one-shot, deterministic classification problem rather than a genuinely progressive, generative task. We propose FlowSG, which recasts SGG as continuous-time transport on a hybrid discrete-continuous state: starting from a noised graph, the model progressively grows an image-conditioned scene graph through constraint-aware refinements that jointly synthesize nodes (objects) and edges (predicates). Specifically, we first leverage a VQ-VAE to quantize a scene graph (e.g., continuous visual features) into compact, predictable tokens; a graph Transformer then (i) predicts a conditional velocity field to transport continuous geometry (boxes) and (ii) updates discrete posteriors for categorical tokens (object features and predicate labels), coupling semantics and geometry via flow-conditioned message aggregation. Training combines flow-matching losses for geometry with a discrete-flow objective for tokens, yielding few-step inference and plug-and-play compatibility with standard detectors and segmenters. Extensive experiments on VG and PSG under closed- and open-vocabulary protocols show consistent gains in predicate R/mR and graph-level metrics, validating the mixed discrete-continuous generative formulation over one-shot classification baselines, with an average improvement of about 3 points over the state-of-the-art USG-Par.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。