AutoHete让大模型训练更快更省力，适配不同硬件自动调参。
AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs
- 根据硬件和需求动态调整存激活、参数和优化器的存放策略。
- 在多种模型规模下比现有系统快1.32至1.91倍。
- 适合资源有限但想训大模型的研究者使用。
基于Transformer的大语言模型（LLMs）在序列建模与文本生成上表现卓越，性能随模型规模提升而增长。然而，GPU显存限制使许多研究者难以进行大模型训练。现有异构训练方法虽显著扩展了可训练模型规模，但引入了大量通信开销和CPU负担。本文提出AutoHete，一个兼容单卡与多卡环境的自动高效异构训练系统。该系统依据具体硬件配置与训练需求，动态调整激活检查点、参数卸载与优化器卸载策略。此外，设计了一种基于优先级的调度机制，最大化跨训练迭代间操作重叠，提升吞吐量。相比最先进异构训练系统，AutoHete在多种模型规模与训练配置下实现1.32x~1.91x的吞吐量提升。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation, with improvements scaling proportionally with model size. However, the limitations of GPU memory have restricted LLM training accessibility for many researchers. Existing heterogeneous training methods significantly expand the scale of trainable models but introduce substantial communication overheads and CPU workloads. In this work, we propose AutoHete, an automatic and efficient heterogeneous training system compatible with both single-GPU and multi-GPU environments. AutoHete dynamically adjusts activation checkpointing, parameter offloading, and optimizer offloading based on the specific hardware configuration and LLM training needs. Additionally, we design a priority-based scheduling mechanism that maximizes the overlap between operations across training iterations, enhancing throughput. Compared to state-of-the-art heterogeneous training systems, AutoHete delivers a 1.32x~1.91x throughput improvement across various model sizes and training configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。