提出新评估框架，发现简单微调比复杂方法更有效
Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols
- 构建10个数据集的基准FEWTRANS，改进评估协议避免过拟合陷阱
- 实测显示预训练模型选择比精调方法更重要，简单全参数微调最优
- 揭示多模态模型在专业领域性能下降源于语言罕见性，提供量化解释
少样本迁移学习因更强的预训练模型和改进的适配算法而快速发展，但缺乏统一、严谨且贴近实际应用的评估协议。本文建立包含10个多样化数据集的FEWTRANS基准，并提出超参数集成（HPE）协议，以克服数据稀缺场景下的“验证集幻觉”。实验表明，预训练模型的选择是性能的主导因素，许多复杂的迁移方法相比简单的全参数微调基线几乎无实际优势。通过深入机制分析发现，全参数微调成功源于分布式微调与高层语义表征的灵活重构，且不易过拟合。此外，我们利用调整后的Zipf频率分数量化了多模态模型在专业领域因语言罕见性导致的性能衰减。通过发布FEWTRANS，旨在为少样本迁移学习研究提供可复现的“标尺”。项目代码已公开于https://github.com/Frankluox/FewTrans。
原文摘要 · Abstract (English)
Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at https://github.com/Frankluox/FewTrans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。