arXiv:2606.06042cs.CV2026-06被引 4

5B参数模型实现高效视频生成与编辑,无需拼接序列

LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing

论文配图:LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing
图 1 · 摘自论文原文
  • 用多模态大语言模型替代文本编码器,结合深度注入对齐特征
  • 零开销缩放加法条件机制,推理速度提升5.41倍
  • 适合电商、时尚等复杂非刚性编辑场景,轻量高效

构建能理解交错多模态输入的统一视频生成与编辑模型是极具前景但充满挑战的领域。现有框架普遍依赖大规模模型(通常130亿参数以上),通过拼接序列令牌引入源视频条件,导致序列长度翻倍,自注意力计算复杂度增加四倍,带来巨大开销。为此,我们提出高效的50亿参数统一架构LoomVideo。LoomVideo将标准文本编码器替换为多模态大语言模型(MLLM),并采用Deepstack注入机制,将多层MLLM特征与扩散变换器(DiT)对齐。关键创新在于提出零开销的缩放加法条件方法:直接将干净源视频潜在表示缩放后加到噪声目标潜在表示上,无需拼接令牌,显著降低计算成本,同时保持对复杂非刚性编辑的强大能力。此外,无缝集成负向时间RoPE策略以处理多参考图像。大量实验表明,该紧凑5B模型在综合基准测试中达到或接近顶尖性能,在电商与时尚生成场景表现尤为突出。得益于零开销条件机制,其推理速度相较同类模型至少提升5.41倍,为实用高效的视频基础模型开辟道路。

原文摘要 · Abstract (English)

Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.

视频生成多模态扩散模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。