一个模型搞定图像生成与理解，支持多种输入和任务。
One Diffusion to Generate Them All
- 将所有任务统一为带噪声序列的训练框架，任意帧可作条件输入。
- 在小数据集上实现文本到图像、多视角生成等任务的优秀表现。
- 适合需要多任务支持、快速个性化和高灵活性的研究者使用。
我们提出 OneDiffusion，一个通用性强的大规模扩散模型，可无缝支持跨多种任务的双向图像生成与理解。它能基于文本、深度图、姿态、布局、语义图等输入进行条件生成，同时处理图像去模糊、超分辨率以及反向任务如深度估计和分割。此外，该模型支持多视角生成、相机位姿估计，并可通过连续图像输入实现即时个性化。模型采用简单而高效的方法：训练时将所有任务视为具有不同噪声尺度的帧序列，推理时任意帧均可作为条件图像。统一训练框架无需专用结构，支持可扩展的多任务训练，且能平滑适应任意分辨率，提升泛化性与可扩展性。实验表明，在相对较小的训练数据集下，其在文本到图像、多视角生成、身份保持、深度估计和相机位姿估计等任务中均达到竞争性性能。代码与模型权重已公开于 https://github.com/lehduong/OneDiffusion。
原文摘要 · Abstract (English)
We introduce OneDiffusion, a versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps, while also handling tasks like image deblurring, upscaling, and reverse processes such as depth estimation and segmentation. Additionally, OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs. Our model takes a straightforward yet effective approach by treating all tasks as frame sequences with varying noise scales during training, allowing any frame to act as a conditioning image at inference time. Our unified training framework removes the need for specialized architectures, supports scalable multi-task training, and adapts smoothly to any resolution, enhancing both generalization and scalability. Experimental results demonstrate competitive performance across tasks in both generation and prediction such as text-to-image, multiview generation, ID preservation, depth estimation and camera pose estimation despite relatively small training dataset. Our code and checkpoint are freely available at https://github.com/lehduong/OneDiffusion
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。