通过特征校准提升合并模型性能,无需重新训练。
FeatCal: Feature Calibration for Post-Merging Models

- 逐层前向校准合并模型权重,减少特征偏移。
- 在CLIP和GLUE上优于现有方法,85.5%准确率领先。
- 仅需少量样本(8例/任务)且计算快4倍,适合实际部署。
模型合并可将多个任务专家模型融合为单一模型,避免联合训练或部署多个模型,但合并后性能常低于原始专家模型。本文通过分析特征漂移——合并模型与专家模型在相同输入下产生的特征差异——揭示性能差距来源。理论表明,特征漂移由上游传播和局部不匹配构成,并随网络深层向前传播累积,最终影响输出表现。基于此,提出FeatCal:利用小规模校准集,按前向顺序逐层校准合并模型权重,降低特征漂移,同时保持接近原始合并权重,保留模型合并优势。该方法采用高效闭式解更新权重,无需梯度下降、迭代优化或额外模块。在主流CLIP和GLUE基准测试中,FeatCal在CLIP-ViT-B/32 Task Arithmetic任务上达到85.5%,优于Surgery(77.0%)和ProbSurgery(78.8%);在FLAN-T5-base GLUE上达85.2%,高于83.7%和82.2%。仅需每任务8个样本即达82.9%准确率,256样本仅耗时53秒,约为基线方法的1/4,体现更高样本效率与更低校准成本。
原文摘要 · Abstract (English)
Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。