arXiv:2604.04407eess.IVcs.CV2026-04被引 1

用未解码语义令牌引导深度超分辨率,提升边界清晰度与跨数据集泛化能力。

NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

论文配图:NAIMA: Semantics Aware RGB Guided Depth Super-Resolution
图 1 · 摘自论文原文
  • 直接注入预训练ViT的深层语义令牌,避免依赖解码后先验
  • 通过注意力机制隐式对齐语义与深度特征,仅用重建损失优化
  • 首次实现无解码语义引导的深度超分,跨数据集性能最强

引导式深度超分辨率(GDSR)是一种多模态方法,利用低分辨率深度图和高分辨率RGB图像恢复更精细的结构细节。然而,RGB图像中误导性的颜色和纹理线索常导致深度图出现伪影和模糊边界。现有方法通过引入大型预训练模型的先验来缓解此问题,但这些先验以相对深度、表面法线或分割图等解码结果形式进入网络,其重建质量受解码精度限制,并需额外辅助目标。本文提出一种新方案:从预训练视觉变压器(ViT)的标记嵌入中生成全局上下文语义先验,直接注入深度分支。提出的引导标记注意力(GTA)模块使多层次深度编码作为查询,在逐步加深的ViT层中跨注意力于语义标记,由零初始化门控调节,仅当语义证据能降低重建误差时才允许其进入。这种标记到深度的对应关系通过单一重建损失下的学习注意力隐式对齐,而非显式对齐或分布匹配目标。基于此,我们提出神经注意力隐式多标记对齐(NAIMA),据我们所知,这是首个由未解码语义标记引导的GDSR框架。NAIMA在同分布下保持竞争力,并实现最强的跨数据集泛化能力。

原文摘要 · Abstract (English)

Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. Recent methods counter this by drawing priors from large pretrained models, but these priors enter the network as decoded predictions such as relative depth, surface normal, or segmentation maps, coupling restoration quality to the accuracy of the decoded prior and requiring auxiliary objectives. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings, injecting them directly into the depth branch. Our Guided Token Attention (GTA) module lets multi-level depth encodings act as queries of cross-attention over semantic tokens drawn from progressively deeper layers of the pretrained visual transformer (ViT), scaled by a zero-initialized gate that admits semantic evidence only to the extent that it reduces reconstruction error. The resulting token-to-depth correspondence is aligned implicitly, through learned attention under a single reconstruction loss, rather than through an explicit alignment or distribution-matching objective. Building on this, we present Neural Attention with Implicit Multi-token Alignment (NAIMA), which, to the best of our knowledge, is the first GDSR framework guided by undecoded semantic tokens. NAIMA remains competitive in-distribution while achieving the strongest cross-dataset generalization.

深度超分语义引导视觉变压器跨域泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。