同时提升视觉语言模型的准确率、泛化能力和抗攻击性。
The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
- 通过优化参数空间曲率和特征空间对齐，统一解决三重权衡问题。
- 在CLIP上实现ID准确率提升10.8%，对抗准确率提升13.5%，OOD保持57.0%。
- 理论驱动的自适应扰动与特征对齐机制，适合构建鲁棒的通用模型。
视觉语言模型（VLMs）微调面临分布内（ID）准确率、分布外（OOD）泛化性和对抗鲁棒性之间的三重权衡。现有方法仅能解决其中两个维度：保泛化方法维持ID/OOD性能但易受对抗攻击；对抗训练虽提升鲁棒性，却降低ID/OOD准确率。本文发现该权衡源于两个几何缺陷：参数空间中的尖锐各向异性极小值，以及扰动下不稳定的特征表示。为此提出GRACE（Gram-aligned Robustness via Adaptive Curvature Estimation），一个联合正则化参数空间曲率与特征空间不变性的统一微调框架。基于鲁棒PAC-Bayes理论，GRACE采用由局部曲率缩放的自适应权重扰动，促进更平坦的极小值，并引入特征对齐损失，保持干净、对抗及分布外输入下的表示一致性。在CLIP模型的ImageNet微调中，GRACE同时实现ID准确率提升10.8%，对抗准确率提升13.5%，并维持57.0%的OOD准确率（对比零样本基线57.4%）。几何分析表明，GRACE收敛至更平坦的极小值且跨分布偏移无特征畸变，为构建基础视觉语言模型的广义鲁棒性提供了原则性进展。
原文摘要 · Abstract (English)
Fine-tuning approaches for Vision-Language Models (VLMs) face a critical three-way trade-off between In-Distribution (ID) accuracy, Out-of-Distribution (OOD) generalization, and adversarial robustness. Existing robust fine-tuning strategies resolve at most two axes of this trade-off. Generalization-preserving methods retain ID/OOD performance but leave models vulnerable to adversarial attacks, while adversarial training improves robustness to targeted attacks but degrades ID/OOD accuracy. Our key insight is that the robustness trade-off stems from two geometric failures: sharp, anisotropic minima in parameter space and unstable feature representations that deform under perturbation. To address this, we propose GRACE (Gram-aligned Robustness via Adaptive Curvature Estimation), a unified fine-tuning framework that jointly regularizes the parameter-space curvature and feature-space invariance for VLMs. Grounded in Robust PAC-Bayes theory, GRACE employs adaptive weight perturbations scaled by local curvature to promote flatter minima, combined with a feature alignment loss that maintains representation consistency across clean, adversarial, and OOD inputs. On ImageNet fine-tuning of CLIP models, GRACE simultaneously improves ID accuracy by 10.8%, and adversarial accuracy by 13.5% while maintaining 57.0% OOD accuracy (vs. 57.4% zero-shot baseline). Geometric analysis confirms that GRACE converges to flatter minima without feature distortion across distribution shifts, providing a principled step toward generalized robustness in foundation VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。