用扩散Transformer提升单目深度估计，效果更准更清晰。
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
- 复用预训练扩散模型，单步推理快速准确
- 在KITTI和ETH3D上绝对误差降低16%-26%
- 能处理毛发、树叶等细长结构，适合实际应用
单目深度估计是计算机视觉中普遍但高度病态的任务，广泛应用于场景重建、计算摄影和机器人等领域。尽管该领域已相当成熟，但现有模型仍难以泛化到分布外输入，且生成的深度图不够锐利细致。本文重新审视Marigold，利用扩散Transformer（DiT）架构，将现代图像生成与编辑模型改造为顶尖的单目深度估计器。我们的方法针对预训练多步流匹配模型进行单步推理，必要时采用量化，保持模型容量的同时运行成本低。我们分析了直接训练带来的伪影，提出两种有效解决方案：对齐模型内部表征与真实深度的语义特征，以及采用基于Sinkhorn的新损失函数构建两阶段微调策略。结果生成的深度图更清晰、更干净，在KITTI和ETH3D上绝对相对误差（AbsRel）相比之前最优模型提升16%-26%。定性上，模型成功还原了毛发、枝叶和极细边缘等此前难以捕捉的细节。此外，Marigold V2在表面法向估计和固有图像分解等其他密集回归任务中也达到当前最优性能。
原文摘要 · Abstract (English)
Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。