GPart通过全局参数分区实现端到端等距微调,大幅降低参数量且保持性能领先。
GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

- 用单一等距映射矩阵将训练向量直接投射到全模型权重空间。
- 仅需d+1个存储值(训练向量加随机种子),参数效率显著优于LoRA。
- 适合追求极致参数效率与简洁架构的LLM微调研究者和应用开发者。
低秩适配(LoRA)已成为大语言模型参数高效微调(PEFT)的主流方法。然而,其双线性结构导致可训练参数到权重更新的映射非保距,扭曲优化空间。尽管如Uni-LoRA等方法通过投影低维向量进入LoRA参数空间提升效率,但后续的双线性映射仍破坏端到端等距性,核心保距问题未解。本文提出GPart(全局参数分区微调),彻底去除低秩瓶颈。该方法使用单个等距划分矩阵,将一个d维可训练向量直接映射至模型全权重空间。结果是极简微调流程:一次随机投影,端到端等距,仅含一个清晰超参数d,存储成本仅为d+1个值(可训练向量加随机种子)。GPart基于理论假设:有效微调可从全权重空间的随机低维子空间中涌现,无需施加低秩矩阵结构。我们在自然语言理解、计算机视觉及数学推理任务上实证表明,GPart性能优于或相当现有PEFT方法。整体上,通过移除结构约束,GPart在效率与性能上达到前沿水平,提供了一条简洁优雅的参数高效微调路径。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has become the dominant paradigm for parameter-efficient fine-tuning (PEFT) of large language models (LLMs). However, its bilinear structure introduces a critical limitation: the mapping from trainable parameters to weight updates is not distance-preserving, distorting the optimization landscape. Methods that project a low-dimensional vector into LoRA's parameter space, such as Uni-LoRA, improve parameter efficiency, but the subsequent bilinear LoRA map breaks end-to-end isometry, leaving the core distance-preservation problem unresolved. We propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method which removes the low-rank bottleneck entirely. Our method uses a single isometric partition matrix to map a $d$-dimensional trainable vector directly into the full weight space of the model. The result is an extremely minimal fine-tuning pipeline: one random projection, end-to-end isometric, with a single clean hyperparameter ($d$) and storage cost of $d+1$ values (the trainable vector plus a random seed). GPart builds on the theoretical premise that effective fine-tuning can emerge from random low-dimensional subspaces of the full weight space, without imposing low-rank matrix structure. We empirically demonstrate the superior or comparable performance of GPart to existing PEFT methods on natural language understanding, computer vision tasks, and mathematical reasoning. Overall, GPart achieves state-of-the-art efficiency and performance by removing structural constraints, offering a straightforward and elegant path to PEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。