通过多项式扩展增强低秩微调，提升大模型表达能力
Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions
- 在低秩空间引入结构化多项式扩展，建模高阶参数交互
- 无需增加秩或推理成本，显著提升多任务性能表现
- 适合需要高效微调且追求高表达能力的场景
低秩微调（LoRA）是高效微调大语言模型的常用方法，但其严格线性结构限制了表达能力。权重更新的双线性形式仅捕捉低秩因子间的一阶依赖，难以建模非线性和高阶参数交互。本文提出多项式扩展低秩适应（PERA），直接在低秩因子空间引入结构化多项式扩展。通过扩展每个低秩因子以合成高阶交互项，再进行组合，使适应空间变为能建模更丰富非线性耦合的多项式流形，且不增加秩或推理开销。理论分析表明，相比现有线性方法，PERA具备更强的表达能力和更优的特征利用效率。实验显示，PERA在多个基准上持续优于当前先进方法。特别地，引入高阶非线性成分（如平方项）对提升表达能力及在不同秩设置下的鲁棒性能至关重要。代码已开源。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a widely used strategy for efficient fine-tuning of large language models (LLMs), but its strictly linear structure fundamentally limits expressive capacity. The bilinear formulation of weight updates captures only first-order dependencies between low-rank factors, restricting the modeling of nonlinear and higher-order parameter interactions. In this paper, we propose Polynomial Expansion Rank Adaptation (PERA), a novel method that introduces structured polynomial expansion directly into the low-rank factor space. By expanding each low-rank factor to synthesize high-order interaction terms before composition, PERA transforms the adaptation space into a polynomial manifold capable of modeling richer nonlinear coupling without increasing rank or inference cost. We provide theoretical analysis demonstrating that PERA offers enhanced expressive capacity and more effective feature utilization compare to existing linear adaptation approaches. Empirically, PERA consistently outperforms state-of-the-art methods across diverse benchmarks. Notably, our experiments show that incorporating high-order nonlinear components particularly square terms is crucial for enhancing expressive capacity and maintaining strong and robust performance under various rank settings. Our code is available at https://github.com/zhangwenhao6/PERA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。