arXiv:2602.09689cs.LG2026-02

仅用一个检查点就能实现模型平均的鲁棒性,无需额外训练。

Model soups need only one ingredient

  • 通过奇异值分解分离参数更新中的关键与噪声成分
  • 自动按层加权,平衡精度与分布外泛化能力
  • 无需数据和超参,可直接用于现有模型

在目标分布上微调大型预训练模型通常能提升分布内(ID)准确率,但会牺牲分布外(OOD)鲁棒性,因表征过度适配微调数据。权重空间集成方法(如Model Soups)虽可缓解此问题,但需训练并存储数十个微调模型,计算成本高昂。本文提出MonoSoup:一种简单、无数据、无超参、事后处理的方法,仅需单个检查点即可实现良好的ID-OOD平衡。该方法对每层更新进行奇异值分解(SVD),分离出高能量方向(捕捉任务特异性适应)与低能量方向(引入噪声但可能保留有用信号)。通过基于熵的有效秩,自动为各组件分配层级权重,考虑模型的谱结构与几何特性。在CLIP模型于ImageNet上微调并在自然分布偏移下评估,以及Qwen语言模型在数学推理与多选基准测试中,该方法表现优异,成为无需多检查点的高效替代方案。

原文摘要 · Abstract (English)

Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data. Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints, but they are computationally prohibitive, requiring the training and storage of dozens of fine-tuned models. In this paper, we introduce MonoSoup, a simple, data-free, hyperparameter-free, post-hoc method that achieves a strong ID-OOD balance using only a single checkpoint. Our method applies Singular Value Decomposition (SVD) to each layer's update and decomposes it into high-energy directions that capture task-specific adaptation and low-energy directions that introduce noise but may still encode residual signals useful for robustness. MonoSoup then uses entropy-based effective rank to automatically re-weigh these components with layer-wise coefficients that account for the spectral and geometric structure of the model. Experiments on CLIP models fine-tuned on ImageNet and evaluated under natural distribution shifts, as well as on Qwen language models tested on mathematical reasoning and multiple-choice benchmarks, show that this plug-and-play approach is a practical and effective alternative to multi-checkpoint methods, retaining much of their benefits without their computational overhead.

模型集成微调鲁棒性SVD

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。