提出新型正交微调方法，效率更高且性能不降。
HOFT: Householder Orthogonal Fine-tuning
- 用豪斯霍尔德变换实现高效正交微调，降低计算开销。
- 在常识推理等任务上达到或超越现有最佳方法。
- 适合追求高泛化性且资源受限的模型适配场景。
使用低秩方法微调基础模型是常见做法。另一种方式是采用正交微调，虽具良好泛化能力，但时间与内存效率较低。本文提出豪斯霍尔德正交微调（HOFT），一种新正交微调方法，旨在缓解时间和空间复杂度问题。进一步理论分析催生了缩放版豪斯霍尔德正交微调（SHOFT）。两者在常识推理、机器翻译、主题驱动生成和数学推理等下游任务中评估，结果表明其性能与当前最优方法相当或更优。
原文摘要 · Abstract (English)
Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。