Liger-Kernel优化LLM训练，提速20%、省60%显存。
Liger Kernel: Efficient Triton Kernels for LLM Training
- 通过操作融合与输入分块技术优化Triton内核
- 相比HuggingFace实现，训练吞吐提升20%，显存减少60%
- 模块化设计，适配各类用户与模型架构
大规模语言模型（LLMs）的高效训练面临巨大挑战，源于其不断增长的计算需求和对性能的更高要求。本文提出Liger-Kernel，一个专为LLM训练设计的开源Triton内核集合。通过内核操作融合与输入分块等优化技术，相比HuggingFace实现，Liger-Kernel在主流LLM上平均提升20%的训练吞吐量，并降低60%的GPU显存占用。该框架注重模块化、可访问性与可扩展性，适用于从初学者到专家的各类用户。内置全面的基准测试与集成验证，确保在多种计算环境和模型架构下具备兼容性、性能、正确性与收敛性。代码已开源，许可证宽松，地址：github.com/linkedin/Liger-Kernel。
原文摘要 · Abstract (English)
Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。