改进低秩微调，让模型更高效地保留输入信息。
IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
- 设计可重构输入的投影框架，避免信息丢失。
- 在常识推理上比LoRA高1.5分，VTAB-1k上高2.3分。
- 参数减半仍达全量微调效果，适合资源受限场景。
参数高效微调（PEFT）方法如LoRA通过向预训练权重注入低秩更新来降低适配成本。然而，LoRA的下投影随机初始化且与数据无关，导致潜在有用信息被丢弃。先前分析表明，该投影在训练中变化很小，而上投影承载了大部分适应信息，因此随机输入压缩成为性能瓶颈。我们提出IPA，一种特征感知的投影框架，旨在在缩减的隐空间中显式重建原始输入。在线性情况下，使用近似前几个主成分的算法实例化IPA，支持高效投影器预训练且推理开销可忽略。在语言和视觉基准测试中，IPA持续优于LoRA和DoRA，常识推理平均提升1.5分，VTAB-1k提升2.3分；当投影冻结时，仅需约一半可训练参数即可达到全量LoRA性能。代码已开源。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。