TLoRA用三矩阵分解实现高效微调，参数量少且性能不降。
TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models
- 将权重更新拆为两个固定随机矩阵和一个可训练矩阵，加层间缩放因子。
- 在GLUE上性能接近LoRA，但可训练参数减少显著。
- 适配动态稳定，逼近LoRA行为，适合资源受限场景。
我们提出TLoRA，一种新颖的三矩阵低秩微调方法，将权重更新分解为两个固定随机矩阵和一个可训练矩阵，并引入可学习的、分层的缩放因子。该三矩阵设计使TLoRA在极少额外计算开销下实现高效的参数适应。在GLUE基准上的大量实验表明，TLoRA性能与现有低秩方法（如LoRA和Adapter）相当，但所需可训练参数显著更少。分析适应动态发现，TLoRA呈现类高斯权重分布、稳定的参数范数及跨层缩放因子变化，进一步凸显其表达能力和适应性。此外，我们发现TLoRA在特征值分布、参数范数和更新余弦相似度上与LoRA高度相似，证明其能有效近似LoRA的适应行为。结果表明，TLoRA是一种高效且有效的大型语言模型微调方法，推动了资源高效模型适配的发展。
原文摘要 · Abstract (English)
We propose TLoRA, a novel tri-matrix low-rank adaptation method that decomposes weight updates into three matrices: two fixed random matrices and one trainable matrix, combined with a learnable, layer-wise scaling factor. This tri-matrix design enables TLoRA to achieve highly efficient parameter adaptation while introducing minimal additional computational overhead. Through extensive experiments on the GLUE benchmark, we demonstrate that TLoRA achieves comparable performance to existing low-rank methods such as LoRA and Adapter-based techniques, while requiring significantly fewer trainable parameters. Analyzing the adaptation dynamics, we observe that TLoRA exhibits Gaussian-like weight distributions, stable parameter norms, and scaling factor variability across layers, further highlighting its expressive power and adaptability. Additionally, we show that TLoRA closely resembles LoRA in its eigenvalue distributions, parameter norms, and cosine similarity of updates, underscoring its ability to effectively approximate LoRA's adaptation behavior. Our results establish TLoRA as a highly efficient and effective fine-tuning method for LLMs, offering a significant step forward in resource-efficient model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。