提出高效长视频生成模型Chimera,解决视觉扩散模型的高计算成本问题。
Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

- 混合架构结合局部卷积与全局注意力,降低计算复杂度。
- 110亿参数模型激活仅20亿参数,训练效率提升7.3倍。
- 无需微调即可从5秒视频泛化到30秒,适合长时序生成任务。
视觉生成对高分辨率图像、长视频和多模态上下文的需求日益增长,但全注意力机制带来的二次方计算开销难以承受。本文提出Chimera,一种混合视觉扩散骨干网络,采用严谨的缩放策略。Chimera以栅格顺序流处理文本、图像和视频标记,无需位置编码,结合Kimi Delta Attention(KDA)实现O(N)复杂度的长序列状态追踪,交错式多头潜在注意力(MLA)支持直接全局交互,以及模态感知的短卷积捕捉局部时空上下文。稀疏专家混合(MoE)层在控制计算量的同时扩展容量。为适配异构架构,提出HeteroP——一种按张量功能输入与模型深度转移超参数的模块级方案,生成一组持续优化的模型,用于拟合符合Chinchilla计算最优规律的激活模型规模、训练标记数及图像-视频数据比。基于这些规律,训练了110亿参数、20亿激活参数的Chimera。实验显示:预训练扩散损失方面,密集骨干比匹配的全注意力Wan-2.1 2B基线高效1.7倍,完整系统达7.3倍;未进行长度微调,模型可零样本外推5秒训练片段至30秒视频,最后5秒仅6.5% FID退化;拟合规律表明,图像预训练中计算资源近乎均分于激活模型规模与训练标记数,而视频预训练在高预算下略偏重模型规模。结果为设计与扩展高效长上下文扩散架构奠定基础。
原文摘要 · Abstract (English)
Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。