AuroOFT让低比特语言模型在保持稳定的同时实现更灵活的非线性微调。
Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning
- 在原有正交微调基础上，添加零初始化的门控非线性残差模块。
- 在1.5B/3B Qwen2.5上提升宏平均准确率1.30%-2.70%，优于QLoRA超6个百分点。
- 适合需要高效微调且追求非线性表达能力的低比特模型部署场景。
量化正交微调（qoft）通过在冻结的量化权重前学习结构化激活旋转，实现了低比特语言模型的参数高效适配。然而，其任务特定更新仅限于线性正交变换，难以实现依赖输入的非线性修正。本文提出AuroOFT，保留qoft作为稳定且兼容量化的方式，同时在每个适配层附加一个零初始化的门控低秩非线性残差。AuroOFT将激活映射至RMS归一化的紧凑潜在空间，使用有界或基于标记的自适应非线性基。零初始化的上投影使AuroOFT在初始化时功能等同于qoft，而正交性仍为分支级稳定性属性，而非整个非线性层的性质。在匹配的数据、优化、解码和解析器协议下，AuroOFT在1.5B/3B Qwen2.5设置上比基准qoft提升宏平均准确率1.30-2.70%，超过QLoRA 6.52-10.62%，且在代表性规模下相比QLoRA减少32.3-44.7%可训练参数。小规模测试题集仅用于协议敏感性诊断。代码已开源。
原文摘要 · Abstract (English)
Quantized orthogonal fine-tuning (qoft) enables parameter-efficient adaptation of low-bit language models by learning structured activation rotations before frozen quantized weights. However, its task-specific updates remain constrained to linear orthogonal transformations, limiting input-dependent nonlinear corrections. We introduce AuroOFT, which keeps qoft as a stable quantization-compatible branch while attaching a zero-start gated low-rank nonlinear residual to each adapted linear layer. AuroOFT maps activations into an RMS-normalized compact latent space and uses adaptive nonlinear bases with bounded or token-dependent gating. The zero-initialized up projection makes AuroOFT functionally identical to qoft at initialization, while orthogonality remains a branch-level stability property rather than a property of the combined nonlinear layer. Under matched data, optimization, decoding, and parser protocols, AuroOFT improves Macro-6 over matched qoft by 1.30-2.70% on the 1.5B/3B Qwen2.5 settings, exceeds QLoRA by 6.52-10.62%, and saves 32.3-44.7% trainable parameters relative to QLoRA in representative scales. The small exam-style multiple-choice math set is treated only as a protocol-sensitivity diagnostic. Our code is available at the anonymous repository: https://anonymous.4open.science/r/AuroOFT-F3FD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。