用块坐标下降法让大模型在普通显卡上低成本训练
Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training
- 用块坐标下降法分批更新参数，降低显存占用
- 70亿模型在RTX 4090上训练成本仅需2.6%原价
- 小团队也能在消费级显卡上跑大模型
大语言模型训练通常需要大量GPU显存和高昂费用，对中小团队构成障碍。本文提出一种基于块坐标下降（BCD）的全参数预训练与微调框架，结合工程优化，可在成本较低的RTX 4090、A100和A800 GPU集群上高效训练大规模模型。在相同硬件条件下，70亿参数模型在A100/A800上的训练成本降至33%，在RTX 4090上仅需2.6%，且性能不降。该方法使原本仅限于A100集群的大模型可在RTX 4090上训练，同时实现与全参数或微调方法相当甚至更优的准确率，显存消耗更低，硬件利用率更高。
原文摘要 · Abstract (English)
Training large language models typically demands extensive GPU memory and substantial financial investment, which poses a barrier for many small- to medium-sized teams. In this paper, we propose a full-parameter pre-training and fine-tuning framework based on block coordinate descent (BCD), enhanced with engineering optimizations, to enable efficient training of large-scale models on cost-effective RTX 4090, A100 and A800 GPU clusters. Under identical hardware configurations, we reduce the training cost of a 7B model to 33% on A100/A800 and only 2.6% on RTX 4090, compared to standard full-parameter training. It also enables large models previously restricted to A100 clusters to be trained on RTX 4090 without degrading performance. BCD achieves comparable or better accuracy than full-parameter and fine-tuning methods at most cases, with lower GPU consumption and improved hardware utilization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。