提出FACT框架，高效实现主动微调，显著提升小样本场景下的模型性能。
FACT: A Simple and Efficient Framework for Active Finetuning
- 分三阶段的层级微调框架，兼顾效率与简单性。
- 低采样率下对ViT模型性能提升超20%，在多个数据集上表现优异。
- 适合标签数据稀缺时的模型优化，尤其适用于大模型小样本场景。
主动微调旨在通过精心挑选的信息量大或具有挑战性的数据，提升预训练模型在特定任务或领域上的表现。以往研究多聚焦于数据选择（即主动部分），但普遍采用全量微调，导致因分布偏移而扭曲预训练特征，尤其当模型规模远大于微调数据量时，过拟合风险加剧。为弥补这一关键缺陷，本文正式定义了FiAF任务，系统探索主动学习中的微调方法。提出FACT框架，一种三阶段层次化微调方案，专为主动微调设计，兼具高效与简洁。实验覆盖三大类数据集：经典（CIFAR10、CIFAR100、ImageNet-1k）、不平衡（CIFAR10-LT、CIFAR100-LT）及细粒度（StanfordCars、FGVCAircraft），每类在3-5种采样比例下评估；采用多种预训练架构（如ConvNeXt、ViT、ViL）；系统研究冻结特征增强（FroFA）策略；并进行全面严谨的效率与泛化能力分析。结果表明，该框架在多项指标上实现显著提升，具备强泛化性和鲁棒性。值得注意的是，在低采样率下，其在ViT模型上对CIFAR10、CIFAR100和ImageNet-1k基准的性能提升超过20%。该系统性方法达到新基准水平，同时保持参数效率，特别适用于标注数据稀缺的场景。
原文摘要 · Abstract (English)
The main goal of active finetuning is to improve a pretrained model's performance on a specific task or domain by finetuning it with carefully selected informative or challenging data. Previous research has predominantly focused on the active aspect (i.e., data selection) while uniformly employing full finetuning for model adaptation, which inevitably distorts pretrained features due to distribution shift. This issue becomes particularly pronounced when the model size is large relative to the finetuning data quantity, leading to heightened overfitting risks. To address this critical gap, we formally outline the FiAF task that emphasizes systematic exploration of finetuning methodologies in active learning. We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity, specifically designed for active finetuning scenarios. Our comprehensive experiments span: (1) Three major dataset categories encompassing classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft) image classification datasets, each evaluated under 3-5 distinct sampling ratios; (2) Diverse pretrained architectures including Convolutional Neural Network (ConvNeXt), Vision Transformer (ViT), and Vision LSTM (ViL) networks; (3) A systematic investigation of frozen feature augmentation (FroFA) strategies. (4) A comprehensive and rigorous analysis of efficiency and generalizability. The results demonstrate significant improvements with strong generalization and robustness. Notably, under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。