通过减少适配层数量实现高效微调，参数减半仍更精准。
FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
- 用单次遍历的费雪得分选关键层，降低训练开销
- 在施蒂费尔流形上约束权重，保持有效秩与正交性
- 在少1/4参数下接近最优模型性能，适合资源受限场景
参数高效微调（PEFT）长期聚焦于LoRA及其精度优化变体，却较少关注原始目标——减少可训练参数。本文提出FoRA，通过减少适配层数量而非降低适配器秩来实现这一目标。FoRA利用单次遍历的对角费雪得分（训练成本低于1%）筛选任务相关层，并在施蒂费尔流形（Stiefel manifold）上训练选定层的LoRA下投影矩阵，以保持列正交性和有效秩。在五个LLaMA系列骨干网络上，FoRA在参数量仅为LoRA和DoRA一半的情况下持续表现更优，在参数量仅为AdaLoRA四分之一时仅落后0.7–0.8准确率点。跨架构实验在来自LLaMA、Qwen3和Gemma系列共十二个骨干网络中验证了从270M到32B参数的稳定提升。两个组件组合具有超加性增益：费雪选择本身即达到同等预算下的秩缩减效果，而施蒂费尔约束带来决定性额外提升。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。