FRoD通过旋转自由度实现高效全秩微调，加速收敛且仅需1.72%参数。
FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence
- 引入旋转自由度与分层联合分解，实现灵活全秩更新。
- 在20个基准上达全模型微调精度，仅用1.72%可训练参数。
- 适合资源受限场景下快速适配大模型的开发者使用。
参数高效微调（PEFT）方法已成为适应大型基础模型到下游任务的实用方案，通过仅更新少量参数降低计算和内存开销。其中，如LoRA等方法试图在效率与表达能力间取得平衡，但常因固有的低秩约束导致收敛缓慢、适应能力有限。为解决这些问题，我们提出FRoD，一种结合分层联合分解与旋转自由度的新微调方法。通过提取跨层全局共享基，并向缩放因子注入稀疏可学习扰动，实现灵活的全秩更新，增强表达能力与效率，带来更快更鲁棒的收敛。在涵盖视觉、推理与语言理解的20个基准上，FRoD在相同训练预算下达到全模型微调精度，仅使用1.72%的可训练参数。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。