统一框架让视频生成模型同时处理多模态输入，支持灵活条件生成。
UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors
- 用随机掩码训练实现任意模态间条件生成，打破固定输入输出限制。
- 在少于1000个视频上训练仍能跨任务泛化，性能媲美顶尖方法。
- 适合需要多模态视频合成的研究者，尤其关注高效通用建模场景。
近期研究表明，视频扩散模型（VDMs）可被用于多种多模态图像任务。然而，现有方法通常为每种任务单独训练模型，固定了输入-输出映射，限制了模态间相关性的建模。本文提出UniVidX，一个基于VDM先验的统一多模态视频生成框架。该框架将像素对齐任务定义为共享多模态空间中的条件生成，适应各模态分布的同时保留骨干模型的原始先验，并在生成中促进跨模态一致性。核心设计包括：随机条件掩码（SCM），在训练时随机划分模态为干净条件与噪声目标，实现全向条件生成；解耦门控LoRA（DGL），为每类模态设置独立LoRA，在其作为生成目标时激活，保护VDM强先验；跨模态自注意力（CMSA），共享键和值但保留模态特定查询，促进信息交换与模态对齐。我们在两个领域实例化：UniVid-Intrinsic，处理RGB视频及其内在图（如反照率、辐照度、法线）；UniVid-Alpha，处理混合RGB视频及其组成RGBA层。实验表明，两者在多个任务上性能均达到或超过当前最优水平，且在仅用不到1000个视频训练下仍具备强鲁棒性与泛化能力。
原文摘要 · Abstract (English)
Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。