用动态提示路由缓解视觉语言模型在长尾数据下的偏见问题
LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions
- 构建五维语义知识库，动态选择最优提示进行微调
- 在CIFAR-LT、ImageNet-LT等数据集上达到当前最优性能
- 计算开销小，适合实际部署的长尾场景微调
预训练视觉语言模型（如CLIP）在视觉任务中表现优异，但微调时常受类别不平衡影响。现有方法虽引入大语言模型补充语义信息，却忽视了预训练阶段固有的类别不平衡，可能导致下游任务中偏差累积。为此，本文提出多维度动态提示路由（MDPR）框架：构建涵盖五个视觉-语义维度的完整知识库，在微调过程中通过动态路由机制对齐全局视觉类别，检索最优提示并平衡细粒度语义，最终通过逻辑值融合实现稳定预测。在CIFAR-LT、ImageNet-LT和Places-LT等长尾基准上的大量实验表明，MDPR性能可媲美当前最先进方法。消融实验进一步验证了语义库对尾部类别的有效性，且动态路由计算开销极低，使MDPR成为应对数据不平衡场景下VLM微调的灵活高效方案。
原文摘要 · Abstract (English)
Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive capability in visual tasks, but their fine-tuning often suffers from bias in class-imbalanced scene. Recent works have introduced large language models (LLMs) to enhance VLM fine-tuning with supplementing semantic information. However, they often overlook inherent class imbalance in VLMs' pre-training, which may lead to bias accumulation in downstream tasks. To address this problem, this paper proposes a Multi-dimensional Dynamic Prompt Routing (MDPR) framework. MDPR constructs a comprehensive knowledge base for classes, spanning five visual-semantic dimensions. During fine-tuning, the dynamic routing mechanism aligns global visual classes, retrieves optimal prompts, and balances fine-grained semantics, yielding stable predictions through logits fusion. Extensive experiments on long-tailed benchmarks, including CIFAR-LT, ImageNet-LT, and Places-LT, demonstrate that MDPR achieves comparable results with current SOTA methods. Ablation studies further confirm the effectiveness of our semantic library for tail classes, and show that our dynamic routing incurs minimal computational overhead, making MDPR a flexible and efficient enhancement for VLM fine-tuning under data imbalance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。