Kron-LoRA用混合分解法让微调更省参数，性能不降反升。
Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning
- 结合克罗内克分解与低秩矩阵压缩，减少参数量
- 比标准LoRA少4倍参数，速度仅慢5-8%
- 适合多任务微调，参数仅需1/4仍保持良好迁移
在多个任务上微调大规模预训练语言模型需要既高效又具备表达力的适配器。我们提出Kron-LoRA，一种将克罗内克结构分解与低秩LoRA压缩相结合的混合适配器——据我们所知，该融合尚未在参数高效微调或矩阵近似文献中被探索。Kron-LoRA相比标准LoRA最多可减少4倍参数，同时保持相近的表达能力。在DistilBERT、Mistral-7B、LLaMA-2-7B和LLaMA-3-8B上，跨越八个基准测试的实验表明，Kron-LoRA在仅带来小幅内存节省的情况下，性能达到或超过LoRA基线，速度仅增加5-8%。在顺序微调场景下，其跨任务迁移表现依然优异，且仅使用四分之一的适配器参数。Kron-LoRA因此为大模型的多任务适配提供了一种可扩展、可持续的解决方案。
原文摘要 · Abstract (English)
Fine-tuning massive pre-trained language models across many tasks demands adapters that are both parameter-efficient and expressive. We introduce \textbf{Kron-LoRA}, a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA compression-an integration that, to our knowledge, has not been explored in parameter-efficient fine-tuning or in matrix approximation literature. Kron-LoRA achieves up to 4$\times$ fewer parameters than standard LoRA while retaining similar expressivity. Experiments on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks show that Kron-LoRA matches or exceeds LoRA baselines with modest memory savings and only a 5-8\% speed overhead. In sequential fine-tuning, it also delivers competitive cross-task transfer despite using only one-quarter of the adapter parameters. Kron-LoRA thus offers a scalable, sustainable solution for multi-task adaptation of large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。