通过动态生成控制参数，让大模型在不同文本条件下自适应调整行为。
Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM
- 用超网络根据文本条件动态生成控制参数β，实现元门控机制。
- 在任务、领域、人设、风格等条件下均优于微调和元学习基线。
- 支持未见任务/条件的泛化，适合需要灵活适配的应用场景。
传统大模型面临语料异质性和细微条件变化的挑战。微调易引发灾难性遗忘，而元学习在大模型上应用受限于复杂性和可扩展性。本文在SwiGLU模块中激活β的元信号，构建自适应非线性调节的元门控机制。采用超网络动态生成β以响应文本条件，赋予大模型元控制能力。在任务、领域、人设、风格等多种条件类型上测试，方法性能优于微调与元学习基线，并可在未见任务、条件类型或指令上合理泛化。代码已开源：https://github.com/AaronJi/MeGan。
原文摘要 · Abstract (English)
Conventional LLMs may suffer from corpus heterogeneity and subtle condition changes. While finetuning can create the catastrophe forgetting issue, application of meta-learning on LLMs is also limited due to its complexity and scalability. In this paper, we activate the meta-signal of $β$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed which dynamically produces $β$ on textual conditions, providing meta-controllability on LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our code can be found in https://github.com/AaronJi/MeGan.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。