用跨模态对比学习，让图像、文本和元数据协同提升细粒度分类效果。
Fine-Grained Classification: Connecting Metadata via Cross-Contrastive Pre-Training
- 通过跨模态对比学习对齐图像、文本与元数据编码器
- 在NABirds数据集上达到84.44%准确率，比基线提升7.83%
- 适合需要融合多源信息的细粒度识别任务
细粒度视觉分类旨在识别超类别下的多个子类，仅靠外观往往难以区分高度相似的类别。本文提出一个统一框架，通过跨对比预训练整合图像、文本和元数据。首先将三种模态编码器对齐至共享嵌入空间，随后微调图像与元数据编码器用于分类。在NABirds数据集上，该方法相较基线提升7.83%，达到84.44%的顶级1准确率，优于现有强大多模态方法。
原文摘要 · Abstract (English)
Fine-grained visual classification aims to recognize objects belonging to many subordinate categories of a supercategory, where appearance alone often fails to distinguish highly similar classes. We propose a unified framework that integrates image, text, and metadata via cross-contrastive pre-training. We first align the three modality encoders in a shared embedding space and then fine-tune the image and metadata encoders for classification. On NABirds, our approach improves over the baseline by 7.83% and achieves 84.44% top-1 accuracy, outperforming strong multimodal methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。