解决视觉语言模型在密集预测中的背景误判问题，提升零样本检测与分割性能。
Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction
- 利用预训练模型检索未标注区域类别，解耦前景与背景特征干扰
- 在Open-Vocabulary检测与分割任务中显著提升性能，优于现有方法
- 无需大量标注，适配多数据集训练，适合零样本场景应用
预训练视觉语言模型（如CLIP）在零样本识别上表现优异，但在密集预测任务中仍表现不佳。自蒸馏近期成为微调此类模型以适应局部区域的有前途方法，但现有先进方法普遍存在显著的‘前景偏差’问题，即模型常将背景区域误判为前景对象。为此，我们提出DenseVLM框架，旨在从强大的预训练视觉语言模型表示中学习无偏的区域-语言对齐。DenseVLM利用预训练模型为未标注区域检索类别，并解耦前景与背景特征间的干扰。我们证明，DenseVLM可直接替代原有视觉语言模型应用于开放词汇目标检测与图像分割方法，带来显著性能提升。此外，其在更大更丰富数据集上训练时展现出良好的零样本可扩展性。代码已开源：https://github.com/HVision-NKU/DenseVLM。
原文摘要 · Abstract (English)
Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot recognition capability, but still underperform in dense prediction tasks. Self-distillation recently is emerging as a promising approach for fine-tuning VLMs to better adapt to local regions without requiring extensive annotations. However, previous state-of-the-art approaches often suffer from significant `foreground bias', where models tend to wrongly identify background regions as foreground objects. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations. DenseVLM leverages the pre-trained VLM to retrieve categories for unlabeled regions and then decouples the interference between foreground and background features. We show that DenseVLM can directly replace the original VLM in open-vocabulary object detection and image segmentation methods, leading to notable performance improvements. Furthermore, it exhibits promising zero-shot scalability when training on more extensive and diverse datasets. Our code is available at https://github.com/HVision-NKU/DenseVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。