arXiv:2509.05661cs.CV2025-09

用语言模型预测未来场景图,让机器理解物体间关系的动态变化。

Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipation

  • 将场景图转为文本序列,用语言模型进行时序关系推理。
  • 在长时序预测中比视觉基线提升21.9%的mR@50指标。
  • 适合需要理解人类行为和交互的智能监控与协作系统。

场景图是动态场景中物体及其时空关系的结构化表示。场景图预测(SGA)旨在从视频片段中预测未来场景图,支持智能监控与人机协作等应用。尽管现有方法能有效利用视觉信息,但长时序预测更依赖语义先验和常识性时间规律,仅靠视觉特征难以提取。为此,我们提出语言驱动的场景图预测(LSGA),将SGA形式化为对文本化场景图序列的时序关系推理,视觉部分由模块化前端处理。在此基础上,提出面向对象的两阶段方法(OOTSM),通过语言模型预测物体集合演化与以物体为中心的关系轨迹,并引入时序一致性正则化。我们在基于Action Genome标注构建的专用基准上评估,结果表明:参数量达30亿的微调语言模型在匹配输入条件下,持续优于GPT-4o、GPT-4o-mini和DeepSeek-V3等零样本/单样本接口。当与现成的视觉场景图生成器结合时,多模态系统在视频基SGA任务上显著提升,长时序预测的mR@50最高提升21.9%。

原文摘要 · Abstract (English)

A scene graph is a structured representation of objects and their spatio-temporal relationships in dynamic scenes. Scene Graph Anticipation (SGA) involves predicting future scene graphs from video clips, enabling applications in intelligent surveillance and human-machine collaboration. While recent SGA approaches excel at leveraging visual evidence, long-horizon forecasting fundamentally depends on semantic priors and commonsense temporal regularities that are challenging to extract purely from visual features. To explicitly model these semantic dynamics, we propose Linguistic Scene Graph Anticipation (LSGA), a linguistic formulation of SGA that performs temporal relational reasoning over sequences of textualized scene graphs, with visual scene-graph detection handled by a modular front-end when operating on video. Building on this formulation, we introduce Object-Oriented Two-Stage Method (OOTSM), a language-based framework that anticipates object-set dynamics and forecasts object-centric relation trajectories with temporal consistency regularization, and we evaluate it on a dedicated benchmark constructed from Action Genome annotations. Extensive experiments show that compact fine-tuned language models with up to 3B parameters consistently outperform strong zero- and one-shot API baselines, including GPT-4o, GPT-4o-mini, and DeepSeek-V3, under matched textual inputs and context windows. When coupled with off-the-shelf visual scene-graph generators, the resulting multimodal system achieves substantial improvements on video-based SGA, boosting long-horizon mR@50 by up to 21.9\% over strong visual SGA baselines.

场景图预测语言模型时序推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。