构建百万级长视频音画数据集,推动开源模型生成高质量电影级内容。
CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation

- 三阶段清洗流程:数据源多样化+电影理论解析+双模态分层标注
- 单个视频平均92.8秒、24.2个镜头,支持多镜头连续叙事生成
- 适配大模型生成电影级音画同步内容,适合影视生成研究者使用
视频生成模型的能力由训练数据的保真度和结构多样性决定。尽管商业系统已能生成电影级叙事,但开源模型受限于高质量数据稀缺。为此,我们推出CineDance-1M,一个大规模开放研究的文本到音画(T2AV)数据集,专为多镜头、长时序联合音画生成设计。每条视频平均92.8秒,含24.2个连续镜头,提供可配置、结构化的音视频标注。通过三阶段严格筛选流程实现高质量:(i) 多源采集与全面清洗,(ii) 借鉴电影理论的叙事解析,(iii) 分层双模态描述。为全面评估,提出CineBench,包含多样化提示集与六维人类对齐指标体系。我们还将LTX-2.3适配至CineDance,其在单模态质量、音画对齐及主体与环境一致性方面表现优异,验证了数据策列策略与数据集质量。本工作有望成为多镜头长视频联合音画生成研究的重要基石。
原文摘要 · Abstract (English)
The fidelity and structural diversity of training datasets fundamentally determine the capabilities of video generation models. While commercial systems showremarkableabilitytogeneratecinematicnarratives, the progress of open-source models remains limited by the scarcity of high-quality training data. To bridge this gap, we introduce CineDance-1M, a large-scale, open research Text-to-Audio-Video (T2AV) dataset designed specifically for multi-shot, long-form joint audio-video generation. Averaging 92.8 seconds and 24.2 continuous shots per video, it provides configurable, structured annotations for both audio and video modalities. This exceptional quality is achieved through a rigorous three-stage curation pipeline: i) diverse sourcing and comprehensive cleansing, ii) film-theory-inspired narrative parsing, and iii) hierarchical dual-modal captioning. For a comprehensive assessment, we propose CineBench, featuring a diverse prompt suite and a six-dimensional, human-aligned metric system tailored for complex narrative audio-video evaluation. Furthermore, we adapt LTX-2.3 into CineDance, which demonstrates exceptional single-modality quality alongside precise audio-video alignment and robust subject and environment consistency, effectively validating our curation strategy and the high quality of CineDance-1M. We anticipate that this work will serve as a solid foundation for accelerating future research in multi-shot, long-form joint audio-video generation. Our project page is available at https://aliothchen.github.io/projects/CineDance/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。