用自然语言瞬间生成适配大模型的LoRA，无需训练
Text-to-LoRA: Instant Transformer Adaption
- 通过超网络在单次前向传播中生成LoRA参数
- 重建的LoRA在多个任务上性能媲美专用适配器
- 支持零样本泛化，适合快速实验与资源有限场景
尽管基础模型为快速内容创作提供了通用工具，但通常需要针对特定任务进行适配。传统方法依赖精心构建的数据集和反复微调底层模型，而微调虽可扩展应用范围，却需昂贵且耗时的训练过程，且对超参数敏感。为此，我们提出Text-to-LoRA（T2L），一种仅凭目标任务的自然语言描述即可即时适配大语言模型的模型。T2L是一个超网络，可在一次低成本前向传播中生成LoRA。在9个预训练LoRA适配器（如GSM8K、Arc）上训练后，我们发现其生成的临时LoRA实例在对应测试集上的表现与专用适配器相当。此外，T2L能压缩数百个LoRA实例，并实现对全新任务的零样本泛化。该方法显著推动了基础模型个性化的普及，实现了低计算开销的语言驱动适配。代码已开源。
原文摘要 · Abstract (English)
While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。