微调图像条件扩散模型，速度提升200倍且精度更高。
Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
- 用单步推理+端到端微调，实现高效深度估计。
- 速度比之前快200倍以上，精度达当前最优水平。
- 方法通用性强，可直接用于Stable Diffusion模型。
近期研究发现，通过将深度估计转化为图像条件图像生成任务，大型扩散模型可作为高精度单目深度估计算法。然而，由于多步推理带来高昂计算成本，限制了其实际应用。本文揭示，这一效率问题源于推理流程中的未被察觉的缺陷。优化后的固定模型在保持最优性能的同时，速度超过200倍。通过在单步模型上使用任务特定损失进行端到端微调，得到一个确定性模型，在常见零样本基准测试中优于所有现有基于扩散模型的深度与法线估计方法。令人意外的是，该微调策略直接应用于Stable Diffusion，也能达到当前最佳性能，挑战了先前部分研究结论的有效性。
原文摘要 · Abstract (English)
Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results, high computational demands due to multi-step inference limited its use in many scenarios. In this paper, we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200$\times$ faster. To optimize for downstream task performance, we perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models on common zero-shot benchmarks. We surprisingly find that this fine-tuning protocol also works directly on Stable Diffusion and achieves comparable performance to current state-of-the-art diffusion-based depth and normal estimation models, calling into question some of the conclusions drawn from prior works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。