通过随机正交变换减少模型融合中的干扰，提升多模型压缩性能。
RanDeS: Randomized Delta Superposition for Multi-Model Compression
- 用随机正交变换打散无关参数差值，避免融合时相互干扰。
- 在视觉与语言任务上显著提升多模型压缩后的性能表现。
- 新增模型无需额外内存，适合动态部署的高效服务场景。
从多模型压缩的角度看，模型融合可实现基于同一基础模型微调的多个模型的内存高效部署，但其性能常因各模型特有参数调整（即delta）间的干扰而下降。本文将模型融合重新定义为一种压缩与检索机制，揭示任务干扰源于检索时无关delta的叠加。为此，我们采用随机正交变换将这些向量去相关，实现自抵消。实验表明，该方法大幅降低干扰，在视觉与语言任务上均显著提升性能。由于变换仅由随机种子决定，新增模型无需额外存储空间。且其对数据与模型无依赖，支持模型增删的低计算开销操作，实现高效灵活的多模型服务。
原文摘要 · Abstract (English)
From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。