基于海森矩阵选择最优微调参数子集，提升效率与泛化能力
Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection
- 利用海森矩阵信息筛选对模型影响最大的参数子集
- 在小于0.1%参数量下实现接近全参数微调的性能
- 适用于不同任务和模型规模，具备跨训练周期迁移性
参数高效微调（PEFT）是一种在保持极低计算开销的前提下，将大型预训练模型适配到下游任务的有效方法。其核心思想是冻结大部分参数，仅训练极小比例的参数子集（如总参数量的<0.1%）。然而，不同方法选择的参数子集不同，导致性能差异显著。这引出一个关键问题：如何有效选择最具影响力的参数子集？本文将子集选择建模为多目标优化问题——在最大化性能的同时最小化可训练参数数量。通过引入ε-约束法与二阶泰勒近似等变换，将其转化为经典的0-1背包问题，并从帕累托最优视角求解。由此提出AdaPEFT，一种基于海森矩阵信息的自适应参数高效微调方法，其选定的参数子集在不同任务、模型规模和训练周期间具有良好的可迁移性。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is a highly effective approach for adapting large pre-trained models to downstream tasks with minimal computational overhead. At the core, PEFT methods freeze most parameters and only trains a small subset (say $<0.1\%$ of total parameters). Notably, different PEFT methods select different subsets, resulting in varying levels of performance. This variation prompts a key question: how to effectively select the most influential subset to train? We formulate the subset selection as a multi-task problem: maximizing the performance and minimizing the number of trainable parameters. We leverage a series of transformations -- including $ε$-constraint method and second-order Taylor approximation -- to arrive at the classical 0-1 knapsack problem, which we solve through the lens of Pareto optimality. Consequently, we propose AdaPEFT, a Hessian-informed PEFT that adapts to various tasks and models, in which the selected subset empirically transfers across training horizons and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。