让单块显卡也能高效微调百亿参数大模型
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
- 用滑动窗口思路实现GPU/CPU/IO异步协同计算
- 峰值内存降低近半，支持8倍更大批量、6倍更大数据集
- 适合资源有限的研究者和开发者快速部署大模型微调
微调大型语言模型（LLM）对领域适应至关重要，但其高内存需求超出多数GPU能力。为解决此问题并推动大模型微调普及，我们提出SlideFormer系统，专为单GPU环境设计。创新包括：(1) 轻量级异步引擎，将GPU视为滑动窗口，实现GPU计算与CPU更新、多级I/O重叠；(2) 高效异构内存管理方案，显著降低峰值内存占用；(3) 优化Triton内核以突破关键瓶颈，并集成先进I/O机制。该协同设计使单张RTX 4090可微调最新123B+模型，支持8倍更大的批量和6倍更大的模型。评估显示，相较于基线，SlideFormer实现1.40x~6.27x更高吞吐量，同时约降低一半的CPU/GPU内存使用，且在NVIDIA和AMD GPU上均保持>95%峰值性能。代码已开源：https://github.com/RegiaYoung/SlideFormer。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.The code is available at https://github.com/RegiaYoung/SlideFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。