改进微调鲁棒性,让模型在数据变化时更稳定。
Revisiting Mixout: An Overlooked Path to Robust Finetuning
- 用动态锚点和可调重采样频率提升微调稳定性
- 在多个数据集上同时提高准确率与抗分布偏移能力
- 仅更新少量参数,适合普通显卡训练
微调视觉基础模型常提升特定数据上的准确率,但会降低分布偏移下的鲁棒性。本文从单次运行、权重共享隐式集成的角度重新审视混入(Mixout),揭示控制鲁棒性的三个关键因素:掩码锚点、重采样频率和掩码稀疏度。基于此,提出GMixout:(i) 用随训练自适应的指数移动平均快照替代固定锚点;(ii) 通过显式超参数调控掩码周期。其稀疏核实现仅更新少量参数,无推理开销,可在消费级显卡上训练。在涵盖协变量偏移、噪声污染和类别不平衡的多个基准测试(ImageNet / ImageNet-LT、DomainNet、iWildCam、CIFAR100-C)上,GMixout不仅超越零样本性能,还在分布偏移下优于Model Soups及主流参数高效微调基线。
原文摘要 · Abstract (English)
Finetuning vision foundation models often improves in-domain accuracy but comes at the cost of robustness under distribution shift. We revisit Mixout, a stochastic regularizer that intermittently replaces finetuned weights with their pretrained reference, through the lens of a single-run, weight-sharing implicit ensemble. This perspective reveals three key levers that govern robustness: the \emph{masking anchor}, \emph{resampling frequency}, and \emph{mask sparsity}. Guided by this analysis, we introduce GMixout, which (i) replaces the fixed anchor with an exponential moving-average snapshot that adapts during training, and (ii) regulates masking period via an explicit resampling-frequency hyperparameter. Our sparse-kernel implementation updates only a small fraction of parameters with no inference-time overhead, enabling training on consumer-grade GPUs. Experiments on benchmarks covering covariate shift, corruption, and class imbalance, ImageNet / ImageNet-LT, DomainNet, iWildCam, and CIFAR100-C, GMixout consistently improves in-domain accuracy beyond zero-shot performance while surpassing both Model Soups and strong parameter-efficient finetuning baselines under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。