让LoRA模型跨不同大模型移植，性能提升4~10%。
TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA
- 通过令牌级对比差异提取任务关键信息，实现参数迁移。
- 在三个基准上平均提升4~10%，无需额外训练模型。
- 适合需要高效迁移LoRA的模型部署场景。
大语言模型广泛应用于实际场景，但微调成本高昂。参数高效微调（PEFT）方法如LoRA可降低开销，但其适配参数依赖基础模型，无法跨模型迁移。现有知识蒸馏方法受限于训练数据，而类似TransLoRA的合成数据生成需额外训练判别器，增加复杂性。本文提出TiTok框架，通过源模型有无LoRA时的令牌级对比差异，捕获任务相关知识，实现无额外模型开销的LoRA移植。该方法可选择性过滤合成数据，提升迁移效率。在多个转移设置下的三个基准测试中，TiTok均表现优异，平均性能较基线提升4~10%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely applied in real world scenarios, yet fine-tuning them comes with significant computational and storage costs. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA mitigate these costs; however, the adapted parameters are dependent on the base model and cannot be transferred across different backbones. One way to address this issue is through knowledge distillation, but its effectiveness inherently depends on training data. Recent work such as TransLoRA avoids this by generating synthetic data; nevertheless, this adds complexity since it requires training an additional discriminator model. In this paper, we propose TiTok, a new framework that enables effective LoRA Transplantation through Token-level knowledge transfer. Specifically, TiTok captures task-relevant information through a token-wise contrastive excess between a source model with and without LoRA. This excess highlights informative tokens and enables selective filtering of synthetic data, all without additional models or overhead. Through experiments on three benchmarks across multiple transfer settings, we demonstrate that TiTok is consistently effective, achieving average performance gains of +4~10% compared to baselines overall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。