用预算引导迭代搜索，自动高效找到小参数量下的最佳微调方案
BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models
- 通过迭代搜索分离模块选择与秩维度搜索，打破传统方法的耦合问题
- 在低参数预算下（如1%），性能超越多数现有PEFT方法
- 适合资源受限场景，尤其适合希望自动化微调的大模型应用者
参数高效微调（PEFT）为下游任务提供了一种高效的大型预训练语言模型微调方案。然而，大多数PEFT策略是手动设计的，常导致性能不佳。近期的自动PEFT方法虽试图解决此问题，但面临搜索空间纠缠、效率低下以及参数预算与搜索过程脱节等挑战。为此，我们提出一种新的预算引导迭代搜索策略（BIPEFT），显著提升搜索效率。BIPEFT采用新型迭代搜索机制，解耦二值模块选择与秩维度搜索空间。同时，基于参数预算设计早期剔除策略，通过逐步移除不重要模块并固定秩维度，加速学习过程。在多个公开基准上的大量实验表明，BIPEFT在极低参数预算下实现了高效且有效的下游任务微调，性能显著优于现有方法。
原文摘要 · Abstract (English)
Parameter Efficient Fine-Tuning (PEFT) offers an efficient solution for fine-tuning large pretrained language models for downstream tasks. However, most PEFT strategies are manually designed, often resulting in suboptimal performance. Recent automatic PEFT approaches aim to address this but face challenges such as search space entanglement, inefficiency, and lack of integration between parameter budgets and search processes. To overcome these issues, we introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT), significantly enhancing search efficiency. BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Additionally, we design early selection strategies based on parameter budgets, accelerating the learning process by gradually removing unimportant modules and fixing rank dimensions. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT in achieving efficient and effective PEFT for downstream tasks with a low parameter budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。