用大模型一键生成长视频章节，准确率超现有方法。
Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs
- 用大模型处理语音转录和帧描述，结合时间戳进行章节划分。
- 在1小时视频上实现45.3的F1分数，远超之前的26.7。
- 轻量级语音引导选帧策略，节省计算资源，适合实用部署。
我们研究视频章节划分任务，即把长视频时间线分割为语义单元并生成对应标题。尽管该任务相对未被充分探索，但自动章节划分可显著提升长视频的导航与内容检索效率。本文提出Chapter-Llama框架，通过在文本域高效解决此问题，在一小时长视频上取得优异表现。具体地，利用具备大上下文窗口的预训练大语言模型（LLM），输入包括语音转录文本和帧描述字幕及其时间戳。针对全帧标注效率低的问题，提出一种基于语音内容的轻量级帧选择策略，实验验证其显著优势。训练模型输出章节边界时间戳及自由形式标题。该方法仅需一次前向传播即可处理一小时视频。在最新VidChapters-7M基准上，性能大幅提升（如F1分数从26.7增至45.3）。为推动后续研究，代码与模型已在项目页面开源。
原文摘要 · Abstract (English)
We address the task of video chaptering, i.e., partitioning a long video timeline into semantic units and generating corresponding chapter titles. While relatively underexplored, automatic chaptering has the potential to enable efficient navigation and content retrieval in long-form videos. In this paper, we achieve strong chaptering performance on hour-long videos by efficiently addressing the problem in the text domain with our 'Chapter-Llama' framework. Specifically, we leverage a pretrained large language model (LLM) with large context window, and feed as input (i) speech transcripts and (ii) captions describing video frames, along with their respective timestamps. Given the inefficiency of exhaustively captioning all frames, we propose a lightweight speech-guided frame selection strategy based on speech transcript content, and experimentally demonstrate remarkable advantages. We train the LLM to output timestamps for the chapter boundaries, as well as free-form chapter titles. This simple yet powerful approach scales to processing one-hour long videos in a single forward pass. Our results demonstrate substantial improvements (e.g., 45.3 vs 26.7 F1 score) over the state of the art on the recent VidChapters-7M benchmark. To promote further research, we release our code and models at our project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。