arXiv:2605.22344cs.CVcs.AI2026-05被引 7

用大模型规划语义,扩散模型生成视频,实现高效精准的视频创作。

Bernini: Latent Semantic Planning for Video Diffusion

论文配图:Bernini: Latent Semantic Planning for Video Diffusion
图 1 · 摘自论文原文
  • 大模型在视觉嵌入空间直接规划视频语义,扩散模型据此生成像素。
  • 在多个视频生成与编辑任务中达到顶尖性能,尤其擅长复杂编辑任务。
  • 支持多模态输入,可分离训练,适合需要强理解与高保真生成的场景。

多模态大语言模型(MLLM)和扩散模型各自发展成熟:前者擅长对异构多模态输入进行推理并具备强语义基础,后者能以照片级真实感合成图像与视频。我们认为两者可通过简单分工统一:由MLLM执行语义规划,扩散模型根据高层语义指导和低层视觉特征生成像素。基于此,我们提出Bernini,一个统一的视频生成与编辑框架。基于MLLM的规划器直接在ViT嵌入空间预测目标语义表示,基于DiT的渲染器则在该计划基础上,结合文本特征和(编辑时)源VAE特征进行像素合成。由于语义作为接口,规划器与渲染器可独立训练,仅需轻度联合训练,保留两者预训练优势的同时保持训练高效。为更好处理多视觉输入,引入分段感知3D旋转位置编码(SA-3D RoPE),并在规划器中加入思维链推理,以增强理解向生成的迁移。Bernini在广泛视频生成与编辑基准上取得最先进性能,其预训练理解能力转化为复杂编辑任务中的强大泛化能力。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.

视频生成扩散模型大模型语义规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。