arXiv:2509.11360cs.CV2025-09被引 2

提出新框架提升视频细粒度描述质量,解决局部与全局信息不一致问题。

GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration

  • 引入视觉专家生成跨帧视觉提示,结合双流结构增强局部描述细节。
  • 通过全局上下文引导局部生成,实现局部与全局描述的动态对齐。
  • 构建新基准和数据集,支持更全面的视频理解评估与训练。

视频细粒度描述旨在生成全面的视频描述以促进视频理解。当前主流方法采用从局部到全局的范式:先生成视频片段的局部描述,再汇总为全局描述。然而,该范式导致描述细节不足且上下文不一致,根源在于(1)缺乏保证细粒度描述的机制,(2)局部与全局描述间交互薄弱。为此,本文提出GLaVE-Cap,一种融合视觉专家的全局-局部对齐视频描述框架,包含两个核心模块:TrackFusion通过视觉专家获取跨帧视觉提示,并采用双流结构实现全面的局部描述生成;CaptionBridge利用全局上下文引导局部生成,并自适应地将局部描述整合为连贯的全局描述。此外,我们构建了GLaVE-Bench,一个涵盖5倍于现有基准每视频查询量、覆盖多种视觉维度的综合性视频描述评测基准;并提供包含16,000条高质量细粒度视频描述及120万问答对的训练数据集GLaVE-1.2M。在四个基准上的大量实验表明,所提方法达到领先性能。消融实验与学生模型分析进一步验证了各模块有效性及GLaVE-1.2M对视频理解社区的贡献。源代码、模型权重、基准与数据集将开源。

原文摘要 · Abstract (English)

Video detailed captioning aims to generate comprehensive video descriptions to facilitate video understanding. Recently, most efforts in the video detailed captioning community have been made towards a local-to-global paradigm, which first generates local captions from video clips and then summarizes them into a global caption. However, we find this paradigm leads to less detailed and contextual-inconsistent captions, which can be attributed to (1) no mechanism to ensure fine-grained captions, and (2) weak interaction between local and global captions. To remedy the above two issues, we propose GLaVE-Cap, a Global-Local aligned framework with Vision Expert integration for Captioning, which consists of two core modules: TrackFusion enables comprehensive local caption generation, by leveraging vision experts to acquire cross-frame visual prompts, coupled with a dual-stream structure; while CaptionBridge establishes a local-global interaction, by using global context to guide local captioning, and adaptively summarizing local captions into a coherent global caption. Besides, we construct GLaVE-Bench, a comprehensive video captioning benchmark featuring 5X more queries per video than existing benchmarks, covering diverse visual dimensions to facilitate reliable evaluation. We further provide a training dataset GLaVE-1.2M containing 16K high-quality fine-grained video captions and 1.2M related question-answer pairs. Extensive experiments on four benchmarks show that our GLaVE-Cap achieves state-of-the-art performance. Besides, the ablation studies and student model analyses further validate the effectiveness of the proposed modules and the contribution of GLaVE-1.2M to the video understanding community. The source code, model weights, benchmark, and dataset will be open-sourced.

视频生成多模态视觉专家描述对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。