arXiv:2409.03583cs.CV2024-09中稿 · BMVC'24, code is a…被引 2

用文本语义关系缓解数据长尾问题,提升小样本类别识别效果。

Text-Guided Mixup Towards Long-Tailed Image Categorization

  • 利用预训练文本编码器的语义关系生成混合样本
  • 在多个长尾数据集上显著提升小类准确率(最高+6.2%)
  • 无需额外标注,适合小样本场景和开放世界分类任务

在许多实际应用中,训练数据的类别分布常呈现长尾特性,这对依赖大量平衡数据的传统深度神经网络训练构成挑战。收集和标注数据以平衡类别分布既耗时又昂贵。现有基于集成学习、重加权或微调的方法在少数类样本稀缺的问题上仍受限。近期研究表明,像CLIP这样的视觉语言模型可通过图像与文本特征间的相似性,有效支持零样本或少样本学习。考虑到大型预训练视觉语言模型可能包含对少数类别有价值的文字信息,本文提出利用文本监督来应对长尾学习挑战。具体地,提出一种新颖的文本引导混合法(text-guided mixup),利用预训练文本编码器识别的类别间语义关系,辅助缓解长尾问题。在多个基准长尾分类任务上的实证研究证明了该方法的有效性,并提供了理论保证。代码已开源:https://github.com/rsamf/text-guided-mixup。

原文摘要 · Abstract (English)

In many real-world applications, the frequency distribution of class labels for training data can exhibit a long-tailed distribution, which challenges traditional approaches of training deep neural networks that require heavy amounts of balanced data. Gathering and labeling data to balance out the class label distribution can be both costly and time-consuming. Many existing solutions that enable ensemble learning, re-balancing strategies, or fine-tuning applied to deep neural networks are limited by the inert problem of few class samples across a subset of classes. Recently, vision-language models like CLIP have been observed as effective solutions to zero-shot or few-shot learning by grasping a similarity between vision and language features for image and text pairs. Considering that large pre-trained vision-language models may contain valuable side textual information for minor classes, we propose to leverage text supervision to tackle the challenge of long-tailed learning. Concretely, we propose a novel text-guided mixup technique that takes advantage of the semantic relations between classes recognized by the pre-trained text encoder to help alleviate the long-tailed problem. Our empirical study on benchmark long-tailed tasks demonstrates the effectiveness of our proposal with a theoretical guarantee. Our code is available at https://github.com/rsamf/text-guided-mixup.

长尾学习视觉语言模型数据增强少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。