无需数据和训练，跨模型迁移LoRA权重，实现快速适配。
Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs
- 通过子空间对齐与投影，实现异构大模型间LoRA模块的无损迁移。
- 在多个常识推理任务上提升基线性能最高达5.26%，接近直接训练效果。
- 全程零数据零训练，20分钟内可在普通显卡完成适配，适合资源受限场景。
传统参数高效微调方法如LoRA与基础模型架构紧密耦合，限制了其在异构预训练大语言模型间的适用性。为此，我们提出Cross-LoRA，一种无需额外训练数据的跨模型LoRA迁移框架。该框架包含两个核心组件：(a) LoRA-Align，通过截断奇异值分解（SVD）与Frobenius最优线性变换，在源模型与目标模型之间进行子空间对齐，解决维度不匹配问题；(b) LoRA-Shift，将对齐后的子空间用于将源模型的LoRA权重更新投影至目标模型参数空间。两者均为数据无关、训练无关，可在消费级GPU上20分钟内完成轻量级适配。在ARCs、OBOA和HellaSwag上的实验表明，Cross-LoRA相较基线模型最高获得5.26%的相对提升；在其他常识推理基准上，性能与直接训练的LoRA适配器相当。
原文摘要 · Abstract (English)
Traditional parameter-efficient fine-tuning (PEFT) methods such as LoRA are tightly coupled with the base model architecture, which constrains their applicability across heterogeneous pretrained large language models (LLMs). To address this limitation, we introduce Cross-LoRA, a data-free framework for transferring LoRA modules between diverse base models without requiring additional training data. Cross-LoRA consists of two key components: (a) LoRA-Align, which performs subspace alignment between source and target base models through rank-truncated singular value decomposition (SVD) and Frobenius-optimal linear transformation, ensuring compatibility under dimension mismatch; and (b) LoRA-Shift, which applies the aligned subspaces to project source LoRA weight updates into the target model parameter space. Both components are data-free, training-free, and enable lightweight adaptation on a commodity GPU in 20 minutes. Experiments on ARCs, OBOA and HellaSwag show that Cross-LoRA achieves relative gains of up to 5.26% over base models. Across other commonsense reasoning benchmarks, Cross-LoRA maintains performance comparable to that of directly trained LoRA adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。