用语言引导贝叶斯优化，30次迭代找到更优LoRA超参。
A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
- 用预训练模型做超参到语义向量的映射，结合自然语言注入领域知识。
- 仅30次迭代即达20%以上性能提升，远超4.5万组合搜索结果。
- 适合需高效微调LLM的科研与工程人员，尤其关注资源受限场景。
使用低秩适应（LoRA）微调大语言模型（LLMs）是一种高效的个性化方法，但其性能高度依赖超参数选择，而穷举搜索计算成本高昂。为此，我们提出一种基于贝叶斯优化（BO）的框架，利用预训练LLM的领域知识来高效搜索LoRA超参数。该方法将预训练LLM用作离散超参数到连续向量空间的映射模块，通过语言提示设计并控制映射过程，以自然语言形式显式注入超参数间的关联与作用机制。同时引入可学习标记捕捉难以用语言描述的残余信息，提升采样效率。此外，利用全量与子集训练间性能强相关性，采用数据子集进行代理训练与评估，显著提升效率。实验表明，仅需约30次迭代即可获得比45,000次组合搜索所得标准超参数高20%以上的性能提升。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) offers a resource-efficient way to personalize or specialize. However, LoRA is highly sensitive to hyperparameter choices, and exhaustive hyperparameter search is computationally expensive. To address this, we propose a Bayesian Optimization (BO) framework that leverages the domain knowledge of pre-trained LLMs to efficiently search for LoRA hyperparameters. Our approach repurposes a pre-trained LLM as a discrete-to-continuous mapping module to link hyperparameters and their domain knowledge to a continuous vector space, where BO is conducted. We design and control the mapping via language prompting, providing a domain-aware textual prompt that describes the relationships among hyperparameters and their respective roles. This allows us to explicitly inject domain knowledge about LoRA into the LLM in natural language. We also introduce an additional learnable token to capture residual information that is difficult to describe linguistically in the prompt. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the strong correlation observed between the performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation using a data subset. This significantly improves the efficiency of our method. We demonstrate that our hyperparameter, discovered with only about 30 iterations, achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations. Project page: https://baekseongeun.github.io/lora-bo/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。