通过双重风险最小化提升微调模型在分布外情况下的鲁棒性
Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
- 用大模型生成核心特征描述，引导零样本预测以估算最坏情况风险
- 在ImageNet、iWildCam等数据集上显著提升分布外性能，最高提升4.7个百分点
- 适合关注模型鲁棒性与实际应用稳定性的研究者和工程师
微调基础模型常导致其对分布偏移的鲁棒性下降。现有方法多致力于保留预训练特征，但未区分特征优劣。本文提出双重风险最小化（DRM），结合经验风险与最坏情况风险最小化，更好保留下游任务的核心特征。具体地，利用大语言模型生成核心特征描述，诱导基于核心的零样本预测，作为最坏情况风险的代理估计。DRM同时优化期望性能与最坏情况性能，在多个真实世界基准上达到新纪录：CLIP ViT-L/14@336在ImageNet上从75.9提升至77.1，WILDS-iWildCam从47.1升至51.8，WILDS-FMoW从50.7升至53.1。代码已开源。
原文摘要 · Abstract (English)
Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9 to 77.1), WILDS-iWildCam (47.1 to 51.8), and WILDS-FMoW (50.7 to 53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。