用图像编辑模型提升单目深度与法线估计,性能超越大模型训练数据百倍的基线。
From Editor to Dense Geometry Estimator
- 将扩散编辑模型改造成几何估计器,通过一致速度损失优化
- 零样本下在ETH3D上深度/法线估计提升超35%,不依赖数据量增长
- 利用全局注意力实现深度与法线联合预测,成本几乎为零
利用预训练文本到图像生成模型的视觉先验在密集预测任务中表现优异。然而,密集预测本质上是图像到图像的任务,提示图像编辑模型可能比生成模型更适合作为基础进行微调。受此启发,我们系统分析了编辑器与生成器在密集几何估计中的微调行为。结果表明,编辑模型具有固有的结构先验,可通过“精炼”其内在特征实现更稳定的收敛,并最终超越生成模型。基于此,我们提出FE2E框架,首次将基于扩散变压器(DiT)架构的先进编辑模型用于密集几何预测。具体而言,为适配该确定性任务,我们将编辑器原有的流匹配损失重构为“一致速度”训练目标,并采用对数量化解决编辑器原生的BFloat16格式与任务高精度需求之间的精度冲突。此外,利用DiT的全局注意力机制,在一次前向传播中免费实现深度与法线的联合估计,使两者的监督信号相互增强。无需扩大训练数据,FE2E在多个数据集上的零样本单目深度与法线估计均取得显著性能提升,尤其在ETH3D上性能提升超过35%,优于经过100倍数据训练的DepthAnything系列。
原文摘要 · Abstract (English)
Leveraging visual priors from pre-trained text-to-image (T2I) generative models has shown success in dense prediction. However, dense prediction is inherently an image-to-image task, suggesting that image editing models, rather than T2I generative models, may be a more suitable foundation for fine-tuning. Motivated by this, we conduct a systematic analysis of the fine-tuning behaviors of both editors and generators for dense geometry estimation. Our findings show that editing models possess inherent structural priors, which enable them to converge more stably by ``refining" their innate features, and ultimately achieve higher performance than their generative counterparts. Based on these findings, we introduce \textbf{FE2E}, a framework that pioneeringly adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction. Specifically, to tailor the editor for this deterministic task, we reformulate the editor's original flow matching loss into the ``consistent velocity" training objective. And we use logarithmic quantization to resolve the precision conflict between the editor's native BFloat16 format and the high precision demand of our tasks. Additionally, we leverage the DiT's global attention for a cost-free joint estimation of depth and normals in a single forward pass, enabling their supervisory signals to mutually enhance each other. Without scaling up the training data, FE2E achieves impressive performance improvements in zero-shot monocular depth and normal estimation across multiple datasets. Notably, it achieves over 35\% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100$\times$ data. The project page can be accessed \href{https://amap-ml.github.io/FE2E/}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。