arXiv:2409.15010cs.CV2024-09IJCAI被引 5

用自回归方式逐步优化单目深度估计,提升精度与泛化能力。

DepthART: Monocular Depth Estimation as Autoregressive Refinement Task

  • 将深度估计转为自回归精修任务,用模型自身预测作为输入迭代优化。
  • 在Hypersim数据集训练后,在多个未见基准上超越现有生成与判别方法。
  • 训练过程动态调整目标,更贴近实际推理,且推理速度更快。

单目深度估计虽在判别式方法上取得进展,但受限于训练数据集。生成式方法通过利用互联网级数据先验已取得显著成效,近期研究显示微调文本到图像扩散模型可达到顶尖性能。然而,自回归生成方法(如视觉自回归模型)在条件图像合成中表现优于扩散模型,且推理更快。本文将视觉自回归变换器(VAR)应用于单目深度估计。但传统基于教师强制的训练方式在深度估计中效果不佳。为此,我们提出DepthART——一种基于模型输出动态构建目标的深度自回归精修训练方法。不同于固定输入与目标的传统训练,本方法以模型自身预测为输入,将目标设为残差最小化,有效缩小训练与推理差异。实验表明,该方法显著提升VAR在深度估计中的性能。在Hypersim数据集上训练后,模型在多个未见基准上优于现有生成与判别基线。

原文摘要 · Abstract (English)

Monocular depth estimation has seen significant advances through discriminative approaches, yet their performance remains constrained by the limitations of training datasets. While generative approaches have addressed this challenge by leveraging priors from internet-scale datasets, with recent studies showing state-of-the-art results using fine-tuned text-to-image diffusion models, there is still room for improvement. Notably, autoregressive generative approaches, particularly Visual AutoRegressive modeling, have demonstrated superior results compared to diffusion models in conditioned image synthesis, while offering faster inference times. In this work, we apply Visual Autoregressive Transformer (VAR) to the monocular depth estimation problem. However, the conventional GPT-2-style training procedure (teacher forcing) inherited by VAR yields suboptimal results for depth estimation. To address this limitation, we introduce DepthART - a novel training method formulated as a Depth Autoregressive Refinement Task. Unlike traditional VAR training with static inputs and targets, our method implements a dynamic target formulation based on model outputs, enabling self-refinement. By utilizing the model's own predictions as inputs instead of ground truth token maps during training, we frame the objective as residual minimization, effectively reducing the discrepancy between training and inference procedures. Our experimental results demonstrate that the proposed training approach significantly enhances the performance of VAR in depth estimation tasks. When trained on Hypersim dataset using our approach, the model achieves superior results across multiple unseen benchmarks compared to existing generative and discriminative baselines.

深度估计自回归视觉生成模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。