发现任务型模型合并会因表征不兼容导致性能崩溃,提出理论解释。
An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse
- 通过实验发现任务表征不兼容是合并失败主因。
- 不同任务组合合并后性能普遍大幅下降,且与参数冲突无关。
- 基于率失真理论建立维度相关合并极限,解释根本约束。
模型合并可将同一基础模型的独立微调版本统一,实现并行开发成果的复用与整合,无需重新训练。然而实践中发现,某些任务专用模型组合在合并后会出现灾难性性能下降,称为合并崩溃。直观上,当不同任务的表征或参数调整存在根本不兼容时,合并会引发破坏性干扰而非协同效应。本文系统识别并刻画了任务级合并崩溃现象:特定任务组合在所有合并方法下均导致显著性能下降。通过大量实验与统计分析,我们发现任务间表征不兼容性与合并崩溃强相关,而参数空间冲突指标相关性极低,挑战了现有模型合并研究中的主流认知。我们基于率失真理论,提出一个维度依赖的理论边界,揭示无论采用何种方法,任务可合并性存在根本限制。
原文摘要 · Abstract (English)
Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。