只优化少数关键维度就能提升多模态微调泛化能力
5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning
- 发现不同参数高效微调方法中存在平坦性偏好
- 仅聚焦少量尖锐维度即可实现优于全量微调的泛化性能
- 提出FlatPO方法专门优化关键尖锐维度，提升模型泛化
参数高效微调（PEFT）方法为将大模型适配到特定领域多模态下游任务提供了高效工具。尽管这些方法在实践中已证明有效，但其核心机制仍缺乏深入探索。本文揭示了各类PEFT方法中普遍存在的平坦性偏好：极少数尖锐维度主导了PEFT的泛化能力。这一发现提示我们，仅关注这些关键尖锐维度，即可获得优于全量微调的泛化效果。为此，我们提出平坦性偏好优化（FlatPO），通过降低这些关键尖锐维度的曲率，使多种PEFT方法实现更优的泛化性能。大量实验验证了该发现与方法的有效性。代码已开源于https://github.com/Can-Lin/FlatPO。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods provide a streamlined and efficient tool for adapting large models to domain-specific multimodal downstream tasks. Although these methods proved their tangible effects in practice, their principal aspects remain under-explored. Therefore we remain curious about the underlying generalization mechanisms in various PEFT methods and how they can be further enhanced. In this paper, we reveal the flatness preference widely present in various PEFTs, where a small fraction of sharp dimensions dominates the generalization of PEFT. This finding suggests an appealing possibility: we may be satisfied with a better generalization by merely attending to this small fraction of sharp dimensions instead of all of them. Furthermore, we propose Flatness Preference Optimization (FlatPO) to flatten these key sharpness dimensions, leading various PEFTs toward better generalization. Extensive experiments demonstrate the effectiveness of our findings and the proposed method. Code is available at https://github.com/Can-Lin/FlatPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。