通过跨关系建模提升细粒度识别,融合多视角提示与多部分视觉特征。
XR-VLM: Cross-Relationship Modeling with Multi-part Prompts and Visual Features for Fine-Grained Recognition
- 设计多部分视觉特征提取与提示学习模块,支持多模态融合。
- 在多个细粒度数据集上超越现有最优方法,显著提升识别精度。
- 适合需要捕捉细微差异的图像分类任务,如物种、车型识别。
视觉语言模型(VLMs)在各类视觉任务中表现优异,但在下游任务中仍需适配才能达到最佳性能。尽管已有多种适配技术提出,但它们在细粒度视觉识别中表现不足,该任务要求模型捕捉细微且具有判别性的特征以区分相似子类别。现有方法通常采用基于对齐的预测框架,即视觉特征与每个类别提示计算相似度作为最终预测,缺乏前向传播中的类别间交互。此外,单一模态特征的学习进一步限制了模型表达能力。为此,我们提出XR-VLM,一种通过建模跨关系发现细微差异的新机制,特别适用于多特征场景。该方法引入统一的多部分视觉特征提取模块,可无缝集成于各类VLM骨干网络;同时开发多部分提示学习模块,捕获子类别的多视角描述。为进一步增强判别力,提出跨关系建模模式,将视觉特征与所有类别提示特征结合,深入探索两模态间的关系。在多个细粒度数据集上的大量实验表明,该方法显著优于当前最先进方法。代码将公开发布。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated impressive performance on various visual tasks, yet they still require adaptation on downstream tasks to achieve optimal performance. Recently, various adaptation technologies have been proposed, but we observe they often underperform in fine-grained visual recognition, which requires models to capture subtle yet discriminative features to distinguish similar sub-categories. Current adaptation methods typically rely on an alignment-based prediction framework, \ie the visual feature is compared with each class prompt for similarity calculation as the final prediction, which lacks class interaction during the forward pass. Besides, learning single uni-modal feature further restricts the model's expressive capacity. Therefore, we propose a novel mechanism, XR-VLM, to discover subtle differences by modeling cross-relationships, which specifically excels in scenarios involving multiple features. Our method introduces a unified multi-part visual feature extraction module designed to seamlessly integrate with the diverse backbones inherent in VLMs. Additionally, we develop a multi-part prompt learning module to capture multi-perspective descriptions of sub-categories. To further enhance discriminative capability, we propose a cross relationship modeling pattern that combines visual feature with all class prompt features, enabling a deeper exploration of the relationships between these two modalities. Extensive experiments have been conducted on various fine-grained datasets, and the results demonstrate that our method achieves significant improvements compared to current state-of-the-art approaches. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。