微调模型的值偏置项，在数据极少时表现最佳。
BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes
- 只微调值投影的偏置项（b_v），实现高效参数更新。
- 在低数据场景下，该方法比微调查询/键偏置性能更高。
- 适用于各类大模型，尤其适合数据稀缺任务。
微调大型语言模型（LLM）的偏置项有望在保持良好性能的同时实现前所未有的参数效率，特别是在数据量有限的情况下。然而，目前尚不清楚微调不同偏置项（即查询、键或值投影中的 $\boldsymbol{b}_q$、$\boldsymbol{b}_k$、$\boldsymbol{b}_v$）与下游任务表现之间的关系。本文系统研究了微调 $\boldsymbol{b}_q$、$\boldsymbol{b}_k$、$\boldsymbol{b}_v$ 与下游任务性能的关联。关键发现表明，在低数据条件下，直接微调 $\boldsymbol{b}_v$ 通常能带来更高的下游性能，优于 $\boldsymbol{b}_q$ 与 $\boldsymbol{b}_k$。我们在涵盖编码器仅和解码器仅架构、最大达 6.7B 参数的多种 LLM 上进行了广泛评估，包括无偏置设计的 LLM。结果为直接微调 $\boldsymbol{b}_v$ 在各类下游任务中的有效性提供了有力证据。代码已公开于 https://github.com/whubaichuan/BEFT。
原文摘要 · Abstract (English)
Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., $\boldsymbol{b}_q$, $\boldsymbol{b}_k$, and $\boldsymbol{b}_v$ in the query, key, or value projections) and downstream performance remains largely unclear to date. In this paper, we investigate the link between fine-tuning $\boldsymbol{b}_q$, $\boldsymbol{b}_k$, and $\boldsymbol{b}_v$ with the performance of the downstream task. Our key finding is that directly fine-tuning $\boldsymbol{b}_v$ generally leads to higher downstream performance in low-data regimes, in comparison to $\boldsymbol{b}_q$ and $\boldsymbol{b}_k$. We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoder-only architectures up to 6.7B parameters (including bias-free LLMs). Our results provide strong evidence for the effectiveness of directly fine-tuning $\boldsymbol{b}_v$ across various downstream tasks. The implementation code is available at https://github.com/whubaichuan/BEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。