用内在通道和多模态控制生成逼真视频，支持局部全局编辑。
X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Rendering
- 基于内在通道与参考图+文本提示的混合控制
- 实现长视频时序一致且保真度高，支持参数化编辑材质光影
- 适合需要精细视频编辑的影视、游戏开发人员
我们提出X2Video，首个可由反照率、法线、粗糙度、金属度和入射光等内在通道引导的扩散模型，用于生成逼真视频，并支持通过参考图像和文本提示对全局与局部区域进行直观多模态控制。内在通道实现对颜色、材质、几何与光照的精准调控，而参考图与文本提示在缺乏内在信息时提供自然调节方式。为此，我们扩展XRGB模型，引入新型高效的混合自注意力机制，保障帧间时序一致性并提升对参考图的保真度；同时设计掩码交叉注意力，有效分离全局与局部文本提示，分别作用于对应区域。针对长视频生成，提出递归采样方法，结合关键帧预测与帧插值，维持长程时序一致性并抑制误差累积。为训练支持，构建包含295个室内场景中1,154个房间的InteriorVideo数据集，提供可靠的内在通道序列与平滑相机轨迹。定性与定量评估均表明，X2Video能生成长时、时序一致、逼真的视频，并有效融合多模态控制，支持通过参数调优对颜色、材质、几何、光照进行联合编辑。
原文摘要 · Abstract (English)
We present X2Video, the first diffusion model for rendering photorealistic videos guided by intrinsic channels including albedo, normal, roughness, metallicity, and irradiance, while supporting intuitive multi-modal controls with reference images and text prompts for both global and local regions. The intrinsic guidance allows accurate manipulation of color, material, geometry, and lighting, while reference images and text prompts provide intuitive adjustments in the absence of intrinsic information. To enable these functionalities, we extend the intrinsic-guided image generation model XRGB to video generation by employing a novel and efficient Hybrid Self-Attention, which ensures temporal consistency across video frames and also enhances fidelity to reference images. We further develop a Masked Cross-Attention to disentangle global and local text prompts, applying them effectively onto respective local and global regions. For generating long videos, our novel Recursive Sampling method incorporates progressive frame sampling, combining keyframe prediction and frame interpolation to maintain long-range temporal consistency while preventing error accumulation. To support the training of X2Video, we assembled a video dataset named InteriorVideo, featuring 1,154 rooms from 295 interior scenes, complete with reliable ground-truth intrinsic channel sequences and smooth camera trajectories. Both qualitative and quantitative evaluations demonstrate that X2Video can produce long, temporally consistent, and photorealistic videos guided by intrinsic conditions. Additionally, X2Video effectively accommodates multi-modal controls with reference images, global and local text prompts, and simultaneously supports editing on color, material, geometry, and lighting through parametric tuning. Project page: https://luckyhzt.github.io/x2video
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。