arXiv:2502.16427cs.CV2025-02ICML被引 7

用图合并方法实现长视频细粒度描述,无需额外训练

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation

  • 将视频分段生成描述,构建场景图再合并为统一图表示
  • 在多个数据集上零样本性能超越现有方法,计算开销更低
  • 适合需要高效处理长视频的视觉语言模型应用

视觉语言模型在图像和短视频描述生成方面取得显著进展,但受限于有限的时间感知范围,难以生成长视频的连贯、全面描述。尽管已有方法通过聚合视频片段信息来缓解此问题,但通常依赖监督微调或带来显著计算开销。为此,我们提出一种基于图合并的长视频描述框架。首先使用现成的视觉描述模型生成各视频片段的描述,再将其解析为独立的场景图,随后合并为统一的图表示,以保留整体上下文与细粒度细节。最后通过轻量级图到文本解码器生成视频级描述。该框架有效扩展了现有模型的时间理解能力,无需在长视频数据集上进行额外微调。实验表明,本方法显著优于现有基于大模型的合并方法,在零样本条件下表现优异且计算成本大幅降低。

原文摘要 · Abstract (English)

Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs.

长视频描述场景图图合并零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。