生成带时间戳的细粒度视频脚本，让机器像编剧一样讲故事。
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
- 提出六维结构化框架，生成带时间戳的影视级视频描述
- 在DailyOmni等任务上超越Gemini-2.5-Pro，提升时序定位与视听推理能力
- 开源42K数据集、7B模型及评估工具，适合视频理解与内容生成研究者
本文提出全景稠密描述（Omni Dense Captioning）新任务，旨在生成连续、细粒度且结构化的音视频叙事，并显式标注时间戳。为实现密集语义覆盖，引入六维结构化模板，生成类似剧本的描述，使读者能逐场景生动还原视频内容。为推动研究，构建高质量人工标注基准OmniDCBench，并提出SodaM统一评估指标，有效缓解场景边界模糊问题。此外，构建包含42,000样本的训练数据集TimeChatCap-42K，提出基于SFT与GRPO训练的TimeChat-Captioner-7B基线模型，使用任务特定奖励。大量实验表明，该模型性能超越Gemini-2.5-Pro，其生成的稠密描述显著提升下游音频-视觉推理（DailyOmni、WorldSense）和时序定位（Charades-STA）能力。所有数据集、模型与代码已开源。
原文摘要 · Abstract (English)
This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are available at https://github.com/yaolinli/TimeChat-Captioner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。