提出轻量动态模型合并方法，用更少参数实现更好性能。
Dynamic Model Merging Made Slim
- 通过可微分秩分配平衡共享与专家参数
- 仅需单个微调模型1.24倍参数即达基准性能
- 无需数据即可恢复任务精度，适合多任务部署
模型合并允许在不联合训练或访问原始数据的情况下复用微调模型。动态合并进一步通过选择性激活相关参数，高效组合多任务专家来提升灵活性。然而，现有方法要么维持完整共享模型搭配微小专家，要么过度分配专家容量，导致准确率-效率权衡不佳。为此，我们提出DiDi-Merging，一种轻量级动态合并框架，利用可微分秩分配平衡共享与专家参数。通过将参数预算建模为低秩模块中的可微分秩优化，并引入无数据精调步骤以恢复任务保真度，DiDi-Merging仅需单个微调模型1.24倍参数即可达到先前动态基线性能，且在1.4倍参数下超越它们，显著低于需超过2倍存储的方法。该方法适用于视觉、语言及多模态任务。
原文摘要 · Abstract (English)
Model merging enables the reuse of fine-tuned models without joint training or access to original data. Dynamic merging further improves flexibility by selectively activating task-relevant parameters and efficiently composing experts across multiple tasks. However, existing dynamic methods either maintain a full shared model with tiny experts or allocate excessive capacity to experts, leading to suboptimal accuracy--efficiency trade-offs. To address this, we propose DiDi-Merging, a slim dynamic merging framework that leverages differentiable rank allocation to balance shared and expert parameters. By formulating parameter budgeting as differentiable rank optimization in low-rank modules and introducing a data-free refinement step to recover task fidelity, DiDi-Merging matches prior dynamic baselines at only 1.24x the parameters of a single fine-tuned model and surpasses them at 1.4x, substantially more compact than methods requiring > 2x storage. DiDi-Merging applies across vision, language, and multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。