用强化学习动态对齐视觉与语言,提升少样本学习的准确率
DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning
- 分层级构建视觉-语言语义,从局部属性到整体描述逐步优化
- 在9个基准上达到新最佳性能,仅用少量样本即可实现高精度识别
- 适合做少样本图像分类、跨模态对齐的研究者参考
少样本学习(FSL)旨在仅用少量样本的情况下泛化到新类别。现有方法虽引入大语言模型(LLMs)通过类别名生成语义嵌入以丰富视觉表征,但忽略了从低层到高层语义的渐进式、自适应对齐,导致语义增益有限。为此,我们提出双层视觉-语言对齐与强化学习门控框架(DVLA-RL),包含双层语义构建(DSC)和强化学习门控注意力(RLA)。DSC将LLM同时作用于类别名和支持样本,生成判别性属性,逐步筛选最相关项并合成连贯的类别描述,提供互补的低层属性与高层描述,实现细粒度定位与整体理解。RLA将跨模态融合建模为序列决策过程,通过轻量级策略使用剧集式REINFORCE训练,动态调整自注意力与交叉注意力的贡献,使浅层聚焦局部属性、深层关注全局语义,实现更精确的跨模态对齐。该方法在三种不同场景下的九个基准上均达到新最优性能。
原文摘要 · Abstract (English)
Few-shot learning (FSL) aims to generalize to novel categories with only a few samples. Recent approaches incorporate large language models (LLMs) to enrich visual representations with semantic embeddings derived from class names. However, they overlook progressive and adaptive alignment between vision and language from low-level to high-level semantics, resulting in limited semantic gains. To address these challenges, we propose Dual-level Vision-Language Alignment with Reinforcement Learning gating (DVLA-RL), which consists of Dual-level Semantic Construction (DSC) and RL-gated Attention (RLA). Specifically, DSC conditions LLMs on both class names and support samples to generate discriminative attributes, progressively selects the most relevant ones, and then synthesizes them into coherent class descriptions. This process provides complementary low-level attributes and high-level descriptions, enabling both fine-grained grounding and holistic class understanding. To dynamically integrate dual-level semantics along with the visual network layers, RLA formulates cross-modal fusion as a sequential decision process. A lightweight policy trained with episodic REINFORCE adaptively adjusts the contributions of self-attention and cross-attention to integrate textual and visual tokens. As a result, shallow layers refine local attributes and deep layers emphasize global semantics, enabling more precise cross-modal alignment. This achieves class-specific discrimination and generalized representations with merely a few support samples. DVLA-RL achieves new state-of-the-art performance across nine benchmarks in three diverse FSL scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。