用单次前向传播将上下文转化为模型参数，实现快速高效微调。
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
- 基于上下文的超网络设计，复用冻结大模型参数生成适配器。
- 单次前向传播生成高质量LoRA，无需微调即可完成复杂问答。
- 相比传统微调节省大量计算资源，适合快速部署与规模化应用。
我们提出SHINE（可扩展的上下文超网络），一种能将多样有意义的上下文映射为高质量LoRA适配器的可扩展超网络。通过在上下文超网络设计中重用冻结大语言模型自身的参数并引入架构创新，SHINE克服了先前超网络的关键局限，在参数量较少的情况下实现了强大的表达能力。我们构建了预训练与指令微调流程，使超网络能在单次前向传播中从多样上下文生成高质量的LoRA适配器。该方法无需任何微调即可更新大模型参数，并能立即支持与上下文相关的复杂问答任务，而无需直接访问上下文，从而在一次前向传播中将上下文知识有效转化为模型参数知识。实验表明，该方法在多种任务上表现优异，相比基于SFT的大模型适配显著节省时间、计算和内存成本，展现出巨大扩展潜力。代码已开源：https://github.com/MuLabPKU/SHINE。
原文摘要 · Abstract (English)
We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。