用文字描述辅助单目深度估计,提升小物体精度与推理速度。
Iris: Integrating Language into Diffusion-based Monocular Depth Estimation
- 利用扩散模型预训练时的语言条件,约束深度解空间。
- 在多个数据集上提升整体精度,尤其改善小区域深度预测。
- 文本越详细,深度结果越精准,且加速训练与推理收敛。
传统单目深度估计受固有歧义和视觉干扰影响。本文证明语言可作为图像外的额外条件,与合理三维场景对齐,从而缩小深度估计的解空间。该条件分布通过扩散模型的文本到图像预训练过程学习。为生成符合文本描述的多视角、多布局图像,模型隐式建模了物体大小、形状、尺度、空间关系及整体场景结构。本文提出 Iris 框架,研究将文本描述融入基于扩散模型的单目深度估计训练与推理。实验采用三种扩散模型(Marigold、Lotus、E2E-FT)及其变体,在 HyperSim 与 Virtual KITTI 上训练,在 NYUv2、KITTI、ETH3D、ScanNet 与 DIODE 上评估。结果表明,该策略显著提升整体深度估计精度,尤其在小物体区域表现更优;文本描述越详细,深度预测可迭代优化。同时,语言作为约束可加速训练与扩散推理轨迹收敛。代码与生成文本数据将在论文接受后公开。
原文摘要 · Abstract (English)
Traditional monocular depth estimation suffers from inherent ambiguity and visual nuisances. We demonstrate that language can enhance monocular depth estimation by providing an additional condition (rather than images alone) aligned with plausible 3D scenes, thereby reducing the solution space for depth estimation. This conditional distribution is learned during the text-to-image pre-training of diffusion models. To generate images under various viewpoints and layouts that precisely reflect textual descriptions, the model implicitly models object sizes, shapes, and scales, their spatial relationships, and the overall scene structure. In this paper, Iris, we investigate the benefits of our strategy to integrate text descriptions into training and inference of diffusion-based depth estimation models. We experiment with three different diffusion-based monocular depth estimators (Marigold, Lotus, and E2E-FT) and their variants. By training on HyperSim and Virtual KITTI, and evaluating on NYUv2, KITTI, ETH3D, ScanNet, and DIODE, we find that our strategy improves the overall monocular depth estimation accuracy, especially in small areas. It also improves the model's depth perception of specific regions described in the text. We find that by providing more details in the text, the depth prediction can be iteratively refined. Simultaneously, we find that language can act as a constraint to accelerate the convergence of both training and the inference diffusion trajectory. Code and generated text data will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。