用少量数据聚类并合并适配器，让手机能运行多任务大模型。
Data-driven Clustering and Merging of Adapters for On-device Large Language Models
- 仅需每任务10个样本，通过迭代优化聚类适配器。
- 在有限存储下，多任务适配器性能显著提升。
- 适合移动端部署，尤其资源受限的设备使用。
移动端大语言模型通常采用任务专用适配器（如LoRA）以在下游任务中实现优异性能。然而，由于内存限制，存储所有适配器不现实；而移动设备通常足以存储有限数量的参数。这引出一个关键挑战：如何选择能在多个任务间良好泛化的代表性适配器——这一问题在现有文献中尚未被探索。我们提出一种名为D2C的新方法，利用最少的任务特定样本（如每任务10个）并采用迭代优化过程来精炼聚类分配。每个簇内的适配器被合并，生成可部署于资源受限设备上的多任务适配器。实验结果表明，该方法在给定存储预算下有效提升了性能。
原文摘要 · Abstract (English)
On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobile devices typically have sufficient capacity to store a limited number of these parameters. This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks - a problem that remains unexplored in existing literature. We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples (e.g., 10 per task) and employs an iterative optimization process to refine cluster assignments. The adapters within each cluster are merged, creating multi-task adapters deployable on resource-constrained devices. Experimental results demonstrate that our method effectively boosts performance for considered storage budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。