用预训练模型同时实现图像生成与深度估计，不损失原有能力。
More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models
- 通过可插拔转换器，在固定模型上无缝切换生成与深度估计模式。
- 在多个基准上达到最优深度估计性能，且保持原图生成功能不变。
- 适合需要多任务统一建模的视觉生成研究者使用。
生成式深度估计方法利用预训练文本到图像扩散模型中丰富的视觉先验，展现出惊人的零样本能力。然而，训练过程中的参数更新会导致模型图像生成能力出现灾难性退化。我们提出MERGE，一种基于固定预训练文本到图像模型的图像生成与深度估计统一模型。MERGE证明，预训练模型不仅能生成图像，还能轻松拓展至深度估计。具体而言，引入一种即插即用框架，通过简单可替换的转换器实现在生成与深度估计模式间的无缝切换；同时提出分组复用机制，促进参数复用并提升额外可学习参数的利用率。MERGE在保留原始图像生成能力的同时，释放了预训练模型强大的深度估计潜力。相较于其他统一模型，MERGE在多个深度估计基准上取得当前最优表现。代码将开源于https://github.com/H-EmbodVis/MERGE。
原文摘要 · Abstract (English)
Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pre-trained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed pre-trained text-to-image model. MERGE demonstrates that the pre-trained text-to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces a play-and-plug framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code will be made available at https://github.com/H-EmbodVis/MERGE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。