让视频生成模型更抗干扰，通过针对性噪声训练提升稳定性。
Corruption-Aware Training of Latent Video Diffusion Models for Robust Text-to-Video Generation
- 设计两类结构化噪声注入方法，保持视频时序一致性。
- 在多个数据集上降低31.9%的FVD，UCF-101提升12.3%。
- 适合追求鲁棒性视频生成或多模态理解的开发者使用。
潜在视频扩散模型（LVDMs）在图像与视频生成中已达顶尖水平，但在噪声条件输入下仍易脆弱，文本或跨模态嵌入的微小扰动会在时间步间累积，引发语义漂移。现有图像扩散中的噪声策略（如高斯、均匀噪声）在视频场景失效，因静态噪声破坏时序连贯性。本文提出CAT-LVDM，一种面向视频扩散的抗干扰训练框架，采用与数据对齐的结构化噪声注入。核心包含两种算子：批处理中心噪声注入（BCNI）和谱感知上下文噪声（SACN），分别对齐批次语义与频谱动态以维持生成一致性。实验显示，BCNI在WebVid-2M、MSR-VTT、MSVD上使FVD降低31.9%，SACN在UCF-101上提升12.3%，优于高斯、均匀噪声及更大规模基线（如DEMO 2.3B、Lavie 3B），且仅用5倍少的数据训练。消融实验证明低秩、数据对齐噪声的独特价值，理论分析进一步揭示其增强鲁棒性与泛化边界的机制。本方法可扩展至自回归生成与多模态视频理解大模型。代码、模型与样本已公开于https://github.com/chikap421/catlvdm。
原文摘要 · Abstract (English)
Latent Video Diffusion Models (LVDMs) have achieved state-of-the-art generative quality for image and video generation; however, they remain brittle under noisy conditioning, where small perturbations in text or multimodal embeddings can cascade over timesteps and cause semantic drift. Existing corruption strategies from image diffusion (Gaussian, Uniform) fail in video settings because static noise disrupts temporal fidelity. In this paper, we propose CAT-LVDM, a corruption-aware training framework with structured, data-aligned noise injection tailored for video diffusion. Our two operators, Batch-Centered Noise Injection (BCNI) and Spectrum-Aware Contextual Noise (SACN), align perturbations with batch semantics or spectral dynamics to preserve coherence. CAT-LVDM yields substantial gains: BCNI reduces FVD by 31.9 percent on WebVid-2M, MSR-VTT, and MSVD, while SACN improves UCF-101 by 12.3 percent, outperforming Gaussian, Uniform, and even large diffusion baselines like DEMO (2.3B) and Lavie (3B) despite training on 5x less data. Ablations confirm the unique value of low-rank, data-aligned noise, and theory establishes why these operators tighten robustness and generalization bounds. CAT-LVDM thus sets a new framework for robust video diffusion, and our experiments show that it can also be extended to autoregressive generation and multimodal video understanding LLMs. Code, models, and samples are available at https://github.com/chikap421/catlvdm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。