让每个输入词独立调整参数，提升低秩微调的表达能力
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
- 为每个输入词动态生成专用投影权重，实现细粒度适配
- 在多个模型和数据集上超越标准LoRA，性能更优
- 无需增加秩就能提升效果，适合资源受限场景
低秩适配（LoRA）是大语言模型中广泛使用的参数高效微调方法，其本质是将输入空间映射到低维输出空间，维度由LoRA秩决定。标准LoRA中所有输入词共享相同权重，执行相同的输入-输出投影，难以捕捉词间语义差异。为此，我们提出面向词粒度的投影低秩适配（TopLoRA），根据输入词动态调整LoRA权重，实现端到端的词粒度输入-输出投影。形式上，其权重为$BΣ_X A$，其中$A$、$B$为低秩矩阵，$Σ_X$由输入词$X$生成的对角矩阵。值得注意的是，TopLoRA不增加原有秩，而是通过学习词粒度的LoRA权重实现更精细适配。大量实验表明，该方法在多个模型与数据集上持续优于标准LoRA及其变体。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as $BΣ_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $Σ_X$ is a diagonal matrix generated from each input token $X$. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。