用小模型快速生成视频场景图,精准且低延迟。
SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

- 采用高效序列化格式与两阶段训练提升生成精度。
- 在多个数据集上实现高精度与约1秒延迟的平衡。
- 适合需要实时结构化视觉理解的应用场景。
场景图生成为视觉感知提供紧凑的结构化表示,但准确快速地从图像和视频中生成场景图仍具挑战。近期基于视觉语言模型的方法可端到端生成结构化文本形式的场景图,但常产生冗长输出并包含无关对象与关系。我们提出SceneGraphVLM,一种基于小型视觉语言模型的图像与视频场景图生成方法。该方法以高效的TOON格式序列化图结构,并采用两阶段训练:监督微调后接强化学习,使用幻觉感知奖励来平衡关系覆盖率与精度,同时惩罚未支持的对象与关系。针对视频,模型可选择性地将前一帧生成的图作为当前帧的轻量级上下文,无需跟踪或后处理即可实现短时记忆。我们在PSG、PVSG和Action Genome数据集上评估了SceneGraphVLM。结合紧凑型VLM与vLLM加速解码,该方法在质量与速度间取得良好权衡,提升了以精度为导向的场景图生成指标,同时保持合理召回率,并实现约1秒的端到端延迟。代码与实现细节见:https://github.com/markus0440/SceneGraphVLM.git。
原文摘要 · Abstract (English)
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。