用低秩元学习让大模型高效适配多个数据集。
Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning
- 在LoRA框架下重定义全局与局部变量，实现大模型元学习。
- 在CrossFit和Unified-QA上准确率与校准误差均优于现有方法。
- 可融合上下文学习，提升法律与化学任务表现。
使用低秩适应（LoRA）微调大语言模型（LLMs）是融入特定数据集信息的低成本方式。但在需融合多个数据集信息的少样本学习场景中，跨数据集泛化能力受限，导致训练成本上升，通常转而采用上下文学习。为此，本文提出一种高效的大模型多分布适应方法——面向LoRA的摊销贝叶斯元学习（ABMLL）。该方法基于小模型的摊销贝叶斯元学习思想，通过重新定义LoRA中的局部与全局变量，并引入新超参数平衡重构精度与任务特异性参数对全局参数的保真度。ABMLL支持在多数据集间有效泛化，可扩展至Llama3-8B与Qwen2-7B等大模型，在CrossFit与Unified-QA数据集上均取得更高的准确率与更低的预期校准误差。此外，元学习可与上下文学习结合，在法律与化学应用中进一步提升性能。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) with low-rank adaptation (LoRA) is a cost-effective way to incorporate information from a specific dataset. However, when a problem requires incorporating information from multiple datasets - as in few shot learning - generalization across datasets can be limited, driving up training costs. As a consequence, other approaches such as in-context learning are typically used in this setting. To address this challenge, we introduce an efficient method for adapting the weights of LLMs to multiple distributions, Amortized Bayesian Meta-Learning for LoRA (ABMLL). This method builds on amortized Bayesian meta-learning for smaller models, adapting this approach to LLMs by reframing where local and global variables are defined in LoRA and using a new hyperparameter to balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. ABMLL supports effective generalization across datasets and scales to large models such as Llama3-8B and Qwen2-7B, outperforming existing methods on the CrossFit and Unified-QA datasets in terms of both accuracy and expected calibration error. We show that meta-learning can also be combined with in-context learning, resulting in further improvements in both these datasets and legal and chemistry applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。