用检索数据做验证,提升少样本识别的泛化能力。
Enabling Validation for Robust Few-Shot Recognition
- 用外部检索数据充当验证集,解决少样本无验证数据难题。
- 新策略使模型在少量标注数据下仍能保持对分布外数据的鲁棒性。
- 适合追求高泛化性的少样本学习研究者和应用开发者。
少样本识别(FSR)通过极少量任务特定标注数据完成分类任务。现有方法通常微调预训练视觉语言模型(VLM),并从开放数据中检索相关但含噪样本以增强训练数据。微调后的VLM在任务内(ID)测试数据上表现良好,但在分布外(OOD)数据上性能下降。本文指出,少样本的核心挑战不仅是训练数据稀缺,更是缺乏验证数据。我们发现一个关键矛盾:检索数据天然与任务内数据分布不同,因此微调后模型在这些数据上表现变差。这导致验证逻辑倾向于不微调的预训练模型,阻碍泛化改进。为解决此困境,提出一种新验证策略,平衡在少数样本任务内数据和检索数据上的性能增益与退化。该策略支持部分微调参数选择与检查点选取,缓解过拟合并提升泛化能力。我们将其与鲁棒学习统一为验证驱动的分阶段微调框架(VEST)。在标准ImageNet OOD基准上大量实验表明,VEST显著优于现有VLM适配方法,在ID与OOD数据上均达到当前最优性能。
原文摘要 · Abstract (English)
Few-Shot Recognition (FSR) tackles classification tasks by training with minimal task-specific labeled data. Prevailing methods adapt or finetune a pretrained Vision-Language Model (VLM) and augment the scarce training data by retrieving task-relevant but noisy samples from open data sources. The finetuned VLM generalizes decently well to the task-specific in-distribution (ID) test data but struggles with out-of-distribution (OOD) test data. This motivates our study of robust FSR with VLM finetuning. The core challenge of FSR is data scarcity, extending beyond limited training data to a complete lack of validation data. We identify a key paradox as a potential solution: repurposing the retrieved open data for validation. As such retrieved data are inherently OOD compared with the task-specific ID training data, finetuned VLMs yield degraded performance on the retrieved data. This causes the validation logic to favor the pretrained model without any finetuning, hindering improvements w.r.t generalization. To resolve this dilemma, we introduce a novel validation strategy that harmonizes performance gain and degradation on the few-shot ID data and the retrieved data, respectively. Our validation enables parameter selection for partial finetuning and checkpoint selection, mitigating overfitting and improving test-data generalization. We unify this strategy with robust learning into a cohesive framework: Validation-Enabled Stage-wise Tuning (VEST). Extensive experiments on the established ImageNet OOD benchmarks show that VEST significantly outperforms existing VLM adaptation methods, achieving state-of-the-art FSR performance on both ID and OOD data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。