无需用户数据,根据任务描述生成个性化LoRA参数。
Semantic-guided LoRA Parameters Generation
- 用任务描述匹配专家任务,生成适配新任务的LoRA参数。
- 零样本下在多任务上表现优于现有方法,支持实时个性化。
- 适合边缘设备上隐私敏感的个性化模型部署场景。
低秩适配(LoRA)在资源受限的边缘设备上高效微调大模型方面展现出强大泛化能力。然而,现实应用中边缘用户常有特定任务偏好,且训练与部署存在显著领域差异,统一模型难以应对。重新为每个用户微调模型又因成本高和数据隐私问题不可行。为此,我们提出首个无需用户数据或额外训练即可生成用户专属LoRA参数的框架——语义引导的LoRA参数生成(SG-LoRA)。SG-LoRA利用任务描述作为语义桥梁,在共享嵌入空间中度量其与一组已知专家任务的相似性,基于此构建目标任务的LoRA参数分布,从而生成高性能参数。该方法通过蒸馏知名LoRA专家知识,实现实时构建符合个体意图的LoRA模型,并在本工作提出的零样本开放世界设定下提供隐私保护的个性化适配方案。多个挑战性任务上的大量实验验证了其卓越性能与强适应性。代码已开源:https://github.com/keepgoingjkg/SG-LoRA。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has demonstrated strong generalization capabilities across a variety of tasks for efficiently fine-tuning AI models, especially on resource-constrained edges. However, in real-world applications, edge users often exhibit task-specific preferences that are difficult to handle with a unified model trained under a closed-world assumption, and the challenge may further increase when there are significant domain shifts between training and deployment. Meanwhile, retraining/fine-tuning models for each user is also impractical due to its cost-intensive nature and privacy concerns over raw data utilization from edges. To address these challenges, we propose Semantic-guided LoRA Parameter Generation (SG-LoRA), the first of its kind framework to efficiently produce user-specific LoRA parameters without any additional training on user tasks or access to user-specific data. Concretely, SG-LoRA uses task descriptions as the semantic bridge, measuring their proximity to a set of known expert tasks in a shared embedding space. Based on this semantic guidance, it models the target task's LoRA parameter distribution to generate high-performing parameters for novel tasks. SG-LoRA enables the real-time construction of LoRA models aligned with individual intents by distilling knowledge from prominent LoRA experts and, meanwhile, offering a privacy-preserving solution for personalized model adaptation in a novel zero-shot open-world setting proposed in this work. Extensive experiments on multiple challenging tasks confirm the superior performance and remarkable adaptability of SG-LoRA. Code is available at https://github.com/keepgoingjkg/SG-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。