arXiv:2412.00054cs.LG2024-12

用二值化开关实现高效多任务模型合并,存储仅需1%-3%。

Less is More: Efficient Model Merging with Binary Task Switch

  • 通过二值化任务向量的符号和开关,压缩参数存储。
  • 在仅1%-3%存储空间下保持接近全精度性能,冗余越多越有效。
  • 适合需要轻量化多任务模型部署的场景,如移动端或边缘设备。

模型合并是一种无需额外训练即可赋予模型多任务能力的有效方法,但现有方法面临参数冗余冲突和存储开销过大的问题。通过受控实验,我们发现任务向量中仅有超过特定阈值的参数对任务有正向贡献,表现出脉冲特性。基于此,我们提出将任务向量二值化以降低存储负担。进一步实验表明,二值化任务向量在微调和合并中几乎无性能损失,且随着冗余参数比例增加,性能提升更显著。据此提出T-Switch:将任务向量分解为三部分——由二值掩码向量实现的激活开关、由二值符号向量实现的极性开关,以及由标量系数实现的缩放旋钮。通过二值化存储任务向量,T-Switch缓解了参数冲突并实现高效存储。为进一步支持自动开关组合,引入Auto-Switch,通过小查询集检索实现无训练组合。实验显示,该方法显著优于现有基线,在仅1%-3%全精度参数存储空间下完成高效合并。

原文摘要 · Abstract (English)

As an effective approach to equip models with multi-task capabilities without additional training, model merging has garnered significant attention. However, existing methods face challenges of redundant parameter conflicts and the excessive storage burden of parameters. In this work, through controlled experiments, we reveal that for task vectors, only those parameters with magnitudes above a certain threshold contribute positively to the task, exhibiting a pulse-like characteristic. We then attempt leveraging this characteristic to binarize the task vectors and reduce storage overhead. Further controlled experiments show that the binarized task vectors incur almost no decrease in fine-tuning and merging performance, and even exhibit stronger performance improvements as the proportion of redundant parameters increases. Based on these insights, we propose Task Switch (T-Switch), which decomposes task vectors into three components: 1) an activation switch instantiated by a binarized mask vector, 2) a polarity switch instantiated by a binarized sign vector, and 3) a scaling knob instantiated by a scalar coefficient. By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage. Furthermore, to enable automated switch combination in T-Switch, we further introduce Auto-Switch, which enables training-free switch combination via retrieval from a small query set. Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.

模型合并二值化多任务轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。