TLoRA+通过优化低秩适配，提升大模型微调性能且不增加计算开销。
TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
- 将TLoRA+优化器融入预训练模型权重矩阵，改进低秩微调机制。
- 在GLUE基准上多架构测试，性能优于传统LoRA且保持高效。
- 适合追求高精度又受限于算力的LLM微调场景。
微调大型语言模型（LLMs）旨在使用相对小规模、特定领域的数据集，将预训练模型适配到具体任务。在参数高效微调（PEFT）方法中，低秩适配（LoRA）因其在不引入额外推理延迟的前提下，达到全量微调性能而脱颖而出。本文提出一种新型PEFT方法，将TLoRA+优化器集成至预训练模型的权重矩阵中。该方法不仅保持了低秩适配的高效性，还进一步提升了性能，且计算成本几乎不变。我们在多种模型架构上的GLUE基准上进行了实验，数值结果一致表明所提方法的有效性和鲁棒性。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。