解决大模型生成长视频字幕长度不足问题,实现超千字精准描述。
LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Models
- 通过分层语义聚合自动生成长视频字幕数据
- 构建10,000条长字幕数据集,使模型输出超1000词
- 新基准测试超越GPT4o等大模型,适合长视频理解任务
大型多模态模型(LMMs)在短视频字幕生成中表现优异,但面对长视频时难以生成详细、完整的长字幕。分析发现,开源LMMs难以持续输出超过300词的文本,导致视觉信息遗漏。我们通过控制实验确认,训练阶段缺乏长字幕配对样本是主要瓶颈。为突破标注成本限制,提出LongCaption-Agent框架,通过分层语义聚合自动合成长字幕数据。基于此构建了新数据集LongCaption-10K,并设计LongCaption-Bench评估基准。将LongCaption-10K用于训练后,模型可生成超过1000词的高质量长字幕,在LongCaption-Bench上性能达到领先水平,甚至优于GPT4o等大型闭源模型。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have demonstrated exceptional performance in video captioning tasks, particularly for short videos. However, as the length of the video increases, generating long, detailed captions becomes a significant challenge. In this paper, we investigate the limitations of LMMs in generating long captions for long videos. Our analysis reveals that open-source LMMs struggle to consistently produce outputs exceeding 300 words, leading to incomplete or overly concise descriptions of the visual content. This limitation hinders the ability of LMMs to provide comprehensive and detailed captions for long videos, ultimately missing important visual information. Through controlled experiments, we find that the scarcity of paired examples with long-captions during training is the primary factor limiting the model's output length. However, manually annotating long-caption examples for long-form videos is time-consuming and expensive. To overcome the annotation bottleneck, we propose the LongCaption-Agent, a framework that synthesizes long caption data by hierarchical semantic aggregation. % aggregating multi-level descriptions. Using LongCaption-Agent, we curated a new long-caption dataset, LongCaption-10K. We also develop LongCaption-Bench, a benchmark designed to comprehensively evaluate the quality of long captions generated by LMMs. By incorporating LongCaption-10K into training, we enable LMMs to generate captions exceeding 1,000 words for long-form videos, while maintaining high output quality. In LongCaption-Bench, our model achieved State-of-The-Art performance, even surpassing larger proprietary models like GPT4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。