通过减去通用知识提升大模型模块化与可复用性
GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction
- 将通用知识与任务专有知识解耦，构建可复用的模块库
- 在多语言基准上实现跨语言性能提升，尤其在弱模型上有效
- 无需额外训练即可动态组合模块，适合快速部署新任务
大语言模型在零样本泛化方面表现不佳，尽管已有多种模块化方法被提出，但我们认为关键瓶颈在于通用知识与任务特定适配的纠缠。为此，我们提出一种模块化框架，通过构建任务专用LoRA模块和通用领域LoRA，从每个任务模块中减去通用知识成分，得到聚焦于任务相关信息的残差模块，称为通用知识减法（GenKnowSub）。利用优化后的任务模块与Arrow路由算法，可在不进行额外训练的情况下动态选择并组合模块以应对新输入。在Phi-3模型及标准Arrow作为基线的实验中，使用英语、法语、德语等多样语言提取的通用知识LoRA，在多种基准测试的单语和跨语言设置下均表现出一致的性能提升。对Phi-2的进一步实验表明，GenKnowSub可推广至能力较弱的大模型。完整代码与数据已公开于https://github.com/saharsamr/Modular-LLM。
原文摘要 · Abstract (English)
Large language models often struggle with zero-shot generalization, and several modular approaches have been proposed to address this challenge. Yet, we hypothesize that a key limitation remains: the entanglement of general knowledge and task-specific adaptations. To overcome this, we propose a modular framework that disentangles these components by constructing a library of task-specific LoRA modules alongside a general-domain LoRA. By subtracting this general knowledge component from each task-specific module, we obtain residual modules that focus more exclusively on task-relevant information, a method we call general knowledge subtraction (GenKnowSub). Leveraging the refined task-specific modules and the Arrow routing algorithm \citep{ostapenko2024towards}, we dynamically select and combine modules for new inputs without additional training. Our studies on the Phi-3 model and standard Arrow as baselines reveal that using general knowledge LoRAs derived from diverse languages, including English, French, and German, yields consistent performance gains in both monolingual and cross-lingual settings across a wide set of benchmarks. Further experiments on Phi-2 demonstrate how GenKnowSub generalizes to weaker LLMs. The complete code and data are available at https://github.com/saharsamr/Modular-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。