arXiv:2505.23209cs.CV2025-05中稿 · ICLR被引 1

提出灵活模型合并框架,按需调节模型大小与精度平衡。

Navigating the Accuracy-Size Trade-Off with Flexible Model Merging

  • 无需数据即可生成不同规模的融合模型,支持多种合并算法
  • 模型规模翻倍时精度最高提升13.5%,超越单一合并模型表现
  • 揭示合并算法在不同规模下排名变化,为模型设计提供新维度

模型合并已成为高效整合多个单任务微调模型的方法。合并后的模型可具备多任务能力,且无需昂贵训练。然而,合并为单一模型常导致精度低于原始微调模型;而部署所有独立模型则带来高存储开销。本文提出FlexMerge——一种无需数据的新型模型合并框架:(a) 可灵活生成从单个合并模型到保留全部微调模型之间任意规模的合并模型;(b) 在统一框架中支持多种合并算法。通过FlexMerge,我们系统刻画了不同算法在精度-尺寸权衡下的表现。研究发现:第一,仅将合并模型规模翻倍,精度即可提升高达13.5%;第二,算法性能排序随模型规模变化,并非始终一致,部分方法在超过单模型规模后反超。这些结果揭示了模型合并的新设计维度:应在全尺度范围内评估和比较算法,而非仅限于单模型极限。在视觉与自然语言处理基准上,涵盖最多30个任务的大量实验验证了FlexMerge的通用性与实用性。

原文摘要 · Abstract (English)

Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to the fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high storage costs. We propose FlexMerge, a novel data-free model merging framework that: (a) flexibly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all fine-tuned models; and (b) supports multiple merging algorithms in a unified framework. Using FlexMerge, we systematically characterize the accuracy-size trade-off of different algorithms. Our study reveals two key findings: first, even modestly larger merged models can yield steep accuracy gains (up to 13.5% when just doubling the size); second, algorithm rankings are not consistent as size increases, with some methods overtaking others beyond the one-model regime. These results uncover a new design dimension for model merging: developing and comparing algorithms across the full spectrum of sizes rather than only at the single-model limit. Extensive experiments on vision and NLP benchmarks, with up to 30 tasks, confirm the generality and practicality of FlexMerge.

模型合并精度优化规模可调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。