用扩散Transformer统一建模图像与深度图，生成更真实且几何合理的结果。
JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers
- 通过自适应调度权重和不平衡时间步采样，实现多模态联合建模。
- 在联合生成、深度估计等任务上表现优异，精度接近条件生成方法。
- 支持多种组合生成任务，只需控制不同分支的时间步即可。
我们提出JointDiT，一种建模RGB与深度联合分布的扩散Transformer。利用先进扩散Transformer的架构优势与出色图像先验，JointDiT不仅能生成高保真图像，还能生成几何合理且准确的深度图。这一稳固的联合分布建模通过两项简单但有效的技术实现：依赖各模态噪声水平的自适应调度权重，以及不平衡时间步采样策略。通过在所有噪声水平下训练每个模态，JointDiT可自然处理多种组合生成任务，包括联合生成、深度估计及深度条件图像生成，仅需控制各分支的时间步即可。实验表明，JointDiT在联合生成上表现卓越；在深度估计与深度条件图像生成任务中也达到可比效果，表明联合分布建模可作为条件生成的可行替代方案。
原文摘要 · Abstract (English)
We present JointDiT, a diffusion transformer that models the joint distribution of RGB and depth. By leveraging the architectural benefit and outstanding image prior of the state-of-the-art diffusion transformer, JointDiT not only generates high-fidelity images but also produces geometrically plausible and accurate depth maps. This solid joint distribution modeling is achieved through two simple yet effective techniques that we propose, namely, adaptive scheduling weights, which depend on the noise levels of each modality, and the unbalanced timestep sampling strategy. With these techniques, we train our model across all noise levels for each modality, enabling JointDiT to naturally handle various combinatorial generation tasks, including joint generation, depth estimation, and depth-conditioned image generation by simply controlling the timesteps of each branch. JointDiT demonstrates outstanding joint generation performance. Furthermore, it achieves comparable results in depth estimation and depth-conditioned image generation, suggesting that joint distribution modeling can serve as a viable alternative to conditional generation. The project page is available at https://byungki-k.github.io/JointDiT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。