用低秩适配与量化技术，让小显存设备也能高效微调金融大模型。
FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation
- 采用量化低秩适配（QLoRA）降低微调计算开销。
- 在金融数据集上实现更高准确率、更低显存占用和更快训练速度。
- 适合需本地部署、资源有限的金融机构使用。
微调的大语言模型在金融任务中表现优异，如情感分析与信息检索。由于隐私考虑，金融机构需本地微调和部署金融大模型（FinLLMs）。然而，微调面临显存不足和长序列处理难题。本文提出量化低秩适配（QLoRA），结合低秩分解与量化技术，显著降低计算需求同时保持高性能。通过数据并行与流水线并行，可在低成本、普及型GPU上实现本地微调。在金融数据集上的实验表明，该方法在准确率、显存使用和训练效率上均有显著提升，验证了低秩方法在可扩展、资源高效微调中的潜力。
原文摘要 · Abstract (English)
Finetuned large language models (LLMs) have shown remarkable performance in financial tasks, such as sentiment analysis and information retrieval. Due to privacy concerns, finetuning and deploying Financial LLMs (FinLLMs) locally are crucial for institutions. However, finetuning FinLLMs poses challenges including GPU memory constraints and long input sequences. In this paper, we employ quantized low-rank adaptation (QLoRA) to finetune FinLLMs, which leverage low-rank matrix decomposition and quantization techniques to significantly reduce computational requirements while maintaining high model performance. We also employ data and pipeline parallelism to enable local finetuning using cost-effective, widely accessible GPUs. Experiments on financial datasets demonstrate that our method achieves substantial improvements in accuracy, GPU memory usage, and time efficiency, underscoring the potential of lowrank methods for scalable and resource-efficient LLM finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。