NNTile让单机训练超大GPT模型成为可能。
NNTile: a machine learning framework capable of training extremely large GPT language models on a single node
- 基于StarPU的任务并行框架，自动调度计算到多核CPU或GPU。
- 实验证明可在单节点上高效训练大规模语言模型。
- 适合需要在单机部署超大模型的研究者与工程师。
本研究提出NNTile框架，用于在异构集群中训练大型深度神经网络。NNTile基于StarPU库，实现任务级并行，将所有任务自动调度至可用的处理单元（CPU和GPU）。这意味着训练大型神经网络所需的特定操作，可由任意CPU核心或GPU设备执行，具体取决于自动调度决策。该方法将计算位置和通信时机的选择从人工决策转移给自动决策系统，无论是简单的贪婪启发式算法还是复杂的基于AI的软件。通过大量数值实验，验证了该工具在训练大型语言模型方面的性能表现。
原文摘要 · Abstract (English)
This study presents an NNTile framework for training large deep neural networks in heterogeneous clusters. The NNTile is based on a StarPU library, which implements task-based parallelism and schedules all provided tasks onto all available processing units (CPUs and GPUs). It means that a particular operation, necessary to train a large neural network, can be performed on any of the CPU cores or GPU devices, depending on automatic scheduling decisions. Such an approach shifts the burden of deciding where to compute and when to communicate from a human being to an automatic decision maker, whether a simple greedy heuristic or a complex AI-based software. The performance of the presented tool for training large language models is demonstrated in extensive numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。