用普通显卡高效训练大模型，7B模型仅需单张16GB显卡。
LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs
- 针对消费级显卡设计优化，支持8比特训练
- 单张16GB显卡可训7B模型，四张4090可训32B模型
- 无需算法近似，性能媲美昂贵云服务器
我们提出LLMQ，一个面向中等规模语言模型（30亿至320亿参数）的端到端CUDA/C++实现，可在价格低廉、配置普通的消费级GPU上进行训练。这类设备内存有限且通信速度慢，因此我们针对这些瓶颈设计了多项优化，包括激活检查点、计算卸载以及基于复制引擎的集合通信。在标准8比特训练流程下，仅使用16GB显存的单张中端游戏显卡即可训练或微调70亿参数模型；配备四张RTX 4090的工作站则可训练320亿参数模型。整个过程无需额外的算法近似，且保持约50%的浮点运算利用率。其效率可与成本更高的云端高性能集群相媲美。
原文摘要 · Abstract (English)
We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are characterized by low memory availability and slow communication compared to datacentre-grade GPUs. Consequently, we showcase a range of optimizations that target these bottlenecks, including activation checkpointing, offloading, and copy-engine based collectives. LLMQ is able to train or fine-tune a 7B model on a single 16GB mid-range gaming card, or a 32B model on a workstation equipped with 4 RTX 4090s. This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%. The efficiency of LLMQ rivals that of production-scale systems on much more expensive cloud-grade GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。