用自动语音转录训练视频大模型，实现低成本实时视频理解。
LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
- 将语音字幕与视频帧按时间对齐，流式训练实现细粒度视觉语言对齐。
- 仅用语音预训练的7B模型即具备实时视频解说能力，超越720亿参数模型。
- 开源500万数据集与完整训练流程，适合追求高效视频理解的研究者。
近期视频大语言模型（Video LLM）通常依赖昂贵的人工标注或专有模型接口（如GPT-4o）生成训练数据，限制了其大规模训练。本文探索利用低成本自动语音识别（ASR）转录进行大规模视频大模型训练。提出一种新型流式训练方法，将ASR词汇与视频帧按时间戳密集交错，自然适配ASR的流式特性，使模型学习到时序对齐的细粒度视觉语言表示。为此构建数据生产流水线，处理YouTube视频及其字幕（同为ASR），生成用于预训练的Live-CC-5M数据集和用于高质量监督微调（SFT）的Live-WhisperX-526K数据集。令人惊讶的是，即使无SFT，仅基于语音预训练的LiveCC-7B-Base模型在通用视频问答任务中表现优异，并展现出实时视频解说的新能力。为此设计新基准LiveSports-3K，使用大模型作为裁判评估自由形式解说质量。实验表明，最终的LiveCC-7B-Instruct模型在实时模式下仍超越先进720亿参数模型（Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B），并在70亿/80亿规模下于VideoMME和OVOBench等主流视频问答基准上达到顶尖水平，证明该方法具有广泛泛化性。所有资源已公开于https://showlab.github.io/livecc。
原文摘要 · Abstract (English)
Recent video large language models (Video LLMs) often depend on costly human annotations or proprietary model APIs (e.g., GPT-4o) to produce training data, which limits their training at scale. In this paper, we explore large-scale training for Video LLM with cheap automatic speech recognition (ASR) transcripts. Specifically, we propose a novel streaming training approach that densely interleaves the ASR words and video frames according to their timestamps. Compared to previous studies in vision-language representation with ASR, our method naturally fits the streaming characteristics of ASR, thus enabling the model to learn temporally-aligned, fine-grained vision-language modeling. To support the training algorithm, we introduce a data production pipeline to process YouTube videos and their closed captions (CC, same as ASR), resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT, the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general video QA performance and exhibits a new capability in real-time video commentary. To evaluate this, we carefully design a new LiveSports-3K benchmark, using LLM-as-a-judge to measure the free-form commentary. Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even working in a real-time mode. Meanwhile, it achieves state-of-the-art results at the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench, demonstrating the broad generalizability of our approach. All resources of this paper have been released at https://showlab.github.io/livecc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。