用深度图辅助预训练,提升手术视觉模型表现
DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models

- 在DINOv2基础上增加像素级深度重建目标
- 8个手术任务上均超越现有基线,尤其提升密集预测性能
- 仅预训练用深度图,推理仍为纯图像,适合临床部署
视觉基础模型(VFMs)在数据稀缺的外科领域极具价值,单个预训练主干网络可为多个下游任务提供丰富表征。然而主流自监督预训练仅使用RGB图像,忽略了可获取的互补信号如深度图。这在手术场景尤为可惜——自然图像模型迁移效果差,而场景几何信息丰富。随着现成模型能为任意图像语料生成伪标注深度图,我们提出DART:一种基于DINOv2的RGB-D预训练方法,通过在掩码iBOT块上加入像素空间深度重建目标,并以伪标签深度监督。深度仅用于预训练,微调与推理仍为纯RGB。实验表明,该像素级重建头提升了表示质量而非破坏它。进一步验证深度(编码场景几何)比边缘等其他密集信号更有效,证明增益源于几何信息本身。在涵盖分割、深度估计和图像级识别的8个外科基准上,DART优于自然图像及领域内基线,包括相同数据下训练的DINOv2,既增强密集预测能力,也提升图像级理解。更广泛而言,DART证明免费可用的几何伪标签可加强基础模型预训练,无需额外标注或增加推理开销,指向更强大的外科骨干网络。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations for many downstream tasks. Yet the dominant self-supervised pretraining paradigm uses only RGB images, leaving readily available complementary signals, such as depth maps, unused. This is a particular missed opportunity in surgery, where natural-image VFMs transfer poorly while the scene geometry is rich and informative. With strong off-the-shelf models now able to produce pseudo-labeled dense depth for any image corpus, we hypothesize that such signals can be folded into pretraining to learn better representations. We present DART, an RGB-D pretraining recipe that builds on DINOv2 with a simple modification: a pixel-space depth reconstruction objective applied to masked iBOT patches, supervised by pseudo-labeled depth. Depth is used only during pretraining, so fine-tuning and inference remain RGB-only. We find that this pixel-level reconstruction head improves representation quality rather than disrupting it. We further show that depth, which encodes scene geometry, is more effective as a target than alternative dense signals such as Canny edges, confirming that the gains stem from depth rather than added supervision alone. Across eight surgical benchmarks spanning segmentation, depth estimation, and image-level recognition, DART outperforms both natural-image and in-domain baselines, including a vanilla DINOv2 trained on identical data, improving dense prediction while also strengthening image-level understanding. More broadly, DART shows that freely available geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。