用非线性变换提升LoRA表达能力，效果优于现有方法。
Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
- 在低秩更新中引入轻量级非线性变换，增强模型表达力。
- 新激活函数Sinter在不增加参数下显著提升性能，优于Sigmoid等常见激活。
- 适用于需要高效微调大模型的场景，尤其适合资源受限环境。
低秩适配（LoRA）是大型语言模型中广泛采用的参数高效微调方法，但其线性特性限制了表达能力。本文提出LoRAN，一种对LoRA的非线性扩展，通过轻量级变换作用于低秩更新。进一步设计了基于正弦的激活函数Sinter，以结构化方式引入扰动，且不增加参数数量。在摘要与分类任务上的实验表明，LoRAN在所有测试中均优于QLoRA。消融研究显示，Sinter在性能上超越Sigmoid、ReLU和Tanh等标准激活函数，凸显激活设计在低秩微调中的关键作用。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。