提出HydraOpt技术，实现适配器合并的高效与高性能自由调节。
HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging
- 利用低秩适配器矩阵间的相似性，动态调节存储与性能平衡。
- 相比独立存储所有适配器，存储量减少48%，性能仅下降0.2%-1.8%。
- 适合移动端等资源受限场景，优于现有合并方法的性能表现。
大语言模型常通过低秩适配器在下游任务中实现强性能，但为每个任务单独存储适配器会显著增加内存开销，对移动设备等资源受限环境构成挑战。尽管模型合并技术可降低存储成本，通常却导致性能大幅下降。本文提出HydraOpt，一种新式模型合并方法，充分利用低秩适配器矩阵间的内在相似性。不同于现有方法固定的效率-性能权衡，HydraOpt可灵活调控该平衡。实验表明，相较于存储所有适配器，HydraOpt将存储量减少48%，性能仅下降0.2%-1.8%，且在相同或略差的存储效率下，优于现有合并技术的性能表现。
原文摘要 · Abstract (English)
Large language models (LLMs) often leverage adapters, such as low-rank-based adapters, to achieve strong performance on downstream tasks. However, storing a separate adapter for each task significantly increases memory requirements, posing a challenge for resource-constrained environments such as mobile devices. Although model merging techniques can reduce storage costs, they typically result in substantial performance degradation. In this work, we introduce HydraOpt, a new model merging technique that capitalizes on the inherent similarities between the matrices of low-rank adapters. Unlike existing methods that produce a fixed trade-off between storage size and performance, HydraOpt allows us to navigate this spectrum of efficiency and performance. Our experiments show that HydraOpt significantly reduces storage size (48% reduction) compared to storing all adapters, while achieving competitive performance (0.2-1.8% drop). Furthermore, it outperforms existing merging techniques in terms of performance at the same or slightly worse storage efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。