arXiv:2505.07652cs.CV2025-05CVPR被引 36

让AI视频生成支持多镜头切换,人物动作连贯一致。

ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models

论文配图:ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
图 1 · 摘自论文原文
  • 引入过渡标记和局部注意力掩码实现镜头分段控制
  • 仅需数千次微调即可生成带镜头指令的多镜头视频
  • 适合需要连续剧情或角色统一的视频创作场景

现有基于扩散模型的文本到视频方法仅能生成单镜头短视频,缺乏生成包含离散切换、同一角色在相同或不同背景中执行不同动作的多镜头视频的能力。为此,我们提出一个框架,包括数据集构建流程和对视频扩散模型的架构扩展,实现文本驱动的多镜头视频生成。该方法将多镜头视频作为整体处理,通过全帧注意力保持角色与背景一致性,并利用镜头特定条件控制镜头数量、时长和内容。关键创新在于引入过渡标记以指定新镜头起始帧,配合局部注意力掩码控制其影响范围,支持针对每个镜头的独立提示。为获取训练数据,我们提出一种新型数据收集管道,从现有单镜头视频数据集中构建多镜头视频数据集。大量实验表明,对预训练文本到视频模型进行数千次迭代微调后,即可实现多镜头视频生成,且性能优于基线方法。

原文摘要 · Abstract (English)

Current diffusion-based text-to-video methods are limited to producing short video clips of a single shot and lack the capability to generate multi-shot videos with discrete transitions where the same character performs distinct activities across the same or different backgrounds. To address this limitation we propose a framework that includes a dataset collection pipeline and architectural extensions to video diffusion models to enable text-to-multi-shot video generation. Our approach enables generation of multi-shot videos as a single video with full attention across all frames of all shots, ensuring character and background consistency, and allows users to control the number, duration, and content of shots through shot-specific conditioning. This is achieved by incorporating a transition token into the text-to-video model to control at which frames a new shot begins and a local attention masking strategy which controls the transition token's effect and allows shot-specific prompting. To obtain training data we propose a novel data collection pipeline to construct a multi-shot video dataset from existing single-shot video datasets. Extensive experiments demonstrate that fine-tuning a pre-trained text-to-video model for a few thousand iterations is enough for the model to subsequently be able to generate multi-shot videos with shot-specific control, outperforming the baselines. You can find more details in https://shotadapter.github.io/

视频生成扩散模型多镜头

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。