构建超大规模视频场景图数据集，助力视频理解模型精准捕捉时空关系。
Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos
- 全自动管道生成636万视频的时空场景图，涵盖660万物体与670万关系。
- 新模型TRaSER单次前向传播生成紧凑场景图，关系/物体识别准确率提升15%-40%。
- 适合视频理解、视觉问答、动作分析等研究者使用，尤其关注时空建模的场景。
我们提出合成视觉知识图谱2（SVG2），一个大规模全景视频场景图数据集。SVG2包含超过636,000个视频，共660万物体、5200万属性和670万关系，规模与多样性较以往时空场景图数据集提升一个数量级。为构建SVG2，我们设计了全自动流水线，融合多尺度全景分割、在线-离线轨迹追踪与自动新对象发现、轨迹语义解析及基于GPT-5的时空关系推断。人工验证显示标注准确率高（物体93.8%，属性88.3%，关系85.4%）。基于此资源，我们训练了TRaSER模型：通过轨迹对齐的标记排列机制、物体轨迹重采样模块与时间窗口重采样模块，将原始视频与全景轨迹一次性转换为紧凑的时空场景图。时间窗口重采样将视觉标记绑定至短轨迹段以保留局部运动与时间语义，物体轨迹重采样则聚合完整轨迹以维持对象全局上下文。在PVSG、VIPSeg、VidOR与SVG2测试集上，TRaSER相较最强开源基线，关系检测提升15%-20%，物体预测提升30%-40%，属性预测提升15%；当其生成的场景图输入视觉语言模型进行视频问答时，相比仅用视频或使用Qwen2.5-VL生成的场景图，准确率绝对提升1.5%-4.6%，证明显式时空场景图作为中间表示的有效性。
原文摘要 · Abstract (English)
We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Human verification of SVG2 annotation accuracy confirms its reliability (objects: 93.8%, attributes: 88.3%, relations: 85.4%). Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。