用统一框架生成与重建4D动态内容，支持单图到全视频任意条件。
One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control
- 通过统一掩码条件机制处理不同稀疏度输入，实现生成与重建无缝切换。
- 在合成与真实4D数据上训练，输出高质量图像和精确点云，保持像素级一致性。
- 采用解耦LoRA控制，避免模型退化，适合几何驱动的4D建模研究者。
我们提出One4D，一个统一的4D生成与重建框架，能同步生成动态4D内容（同步的RGB帧与点云图）。通过统一掩码条件（UMC）机制，一致处理不同稀疏度的条件帧，使模型可无缝切换于单图生成、全视频重建以及稀疏帧混合生成与重建任务。框架基于强大视频生成模型，设计了专门的网络结构以联合生成RGB帧与点云图。现有扩散微调策略在联合生成任务中常导致深度图或点云重建失败，并快速损害基础视频模型性能。为此，我们引入解耦LoRA控制（DLC），使用两个模态专用的LoRA适配器构建解耦计算分支，通过轻量级、零初始化的控制连接逐步学习跨模态像素级一致性。在适度计算预算下，结合合成与真实4D数据集进行训练，One4D在生成与重建任务中均产出高质量的RGB帧与精确点云。本工作推动了基于视频扩散模型的通用、高质量几何4D世界建模。
原文摘要 · Abstract (English)
We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。