给模型合并加层敏感度调节,提升多任务性能
LARV: Data-Free Layer-wise Adaptive Rescaling Veneer for Model Merging
- 按层自适应调整任务向量权重,抑制浅层干扰
- 在多个视觉模型上提升合并效果,最高达92.6%
- 无需训练数据和重训,可无缝接入现有合并方法
模型合并旨在不依赖训练数据的情况下,将多个微调模型融合为单一多任务模型。现有任务向量合并方法(如TIES、TSV-M、Iso-C/CTS)虽聚合规则不同,但基本对所有层采用统一处理,忽略了大型视觉变换器中各层的显著差异:浅层对干扰敏感,深层则编码稳定任务特征。本文提出LARV——一种无需训练、无数据、与合并方式无关的分层自适应缩放外罩,可插入任意任务向量合并器,在聚合前为每个任务向量分配逐层缩放系数。LARV通过简单确定性调度自适应抑制浅层干扰、增强深层对齐,无需重训或修改原有合并器。据我们所知,这是首个针对任务向量合并进行分层缩放的工作。LARV仅需计算数据无关的层代理指标,并通过轻量规则生成缩放值;我们研究了多种实例化方案(如固定层级两/三级缩放、连续映射),发现层级选择具最优鲁棒性,而连续映射仅为消融实验。LARV与基础合并器正交,开销极小。在FusionBench上,使用视觉变换器时,LARV在8/14/20任务设置下持续提升所有任务向量基线,例如Iso-C + LARV在ViT-B/32上达85.9%,ViT-B/16上达89.2%,ViT-L/14上达92.6%。分层分析与扰动测试表明,LARV有效抑制浅层干扰,适度增强深层任务稳定特征,使模型合并从均匀过程变为稳健的分层过程。
原文摘要 · Abstract (English)
Model merging aims to combine multiple fine-tuned models into a single multi-task model without access to training data. Existing task-vector merging methods such as TIES, TSV-M, and Iso-C/CTS differ in their aggregation rules but treat all layers nearly uniformly. This assumption overlooks the strong layer-wise heterogeneity in large vision transformers, where shallow layers are sensitive to interference while deeper layers encode stable task-specific features. We introduce LARV, a training-free, data-free, merger-agnostic Layer-wise Adaptive Rescaling Veneer that plugs into any task-vector merger and assigns a per-layer scale to each task vector before aggregation, and show it consistently boosts diverse merging rules. LARV adaptively suppresses shallow-layer interference and amplifies deeper-layer alignment using a simple deterministic schedule, requiring no retraining or modification to existing mergers. To our knowledge, this is the first work to perform layer-aware scaling for task-vector merging. LARV computes simple data-free layer proxies and turns them into scales through a lightweight rule; we study several instantiations within one framework (e.g., tiered two/three-level scaling with fixed values, or continuous mappings) and show that tiered choices offer the best robustness, while continuous mappings remain an ablation. LARV is orthogonal to the base merger and adds negligible cost. On FusionBench with Vision Transformers, LARV consistently improves all task-vector baselines across 8/14/20-task settings; for example, Iso-C + LARV reaches 85.9% on ViT-B/32, 89.2% on ViT-B/16, and 92.6% on ViT-L/14. Layerwise analysis and corruption tests further indicate that LARV suppresses shallow-layer interference while modestly amplifying deeper, task-stable features, turning model merging into a robust, layer-aware procedure rather than a uniform one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。