ChunkFT让大模型在单张显卡上高效全参数微调，节省大量显存。
ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
- 按需计算梯度子块，不改网络结构即可实现任意子网络优化
- 70亿参数模型微调仅需13.72GB显存，单张4090可运行80亿模型
- 性能接近全参数微调，适合资源有限却想做完整微调的研究者
本文提出ChunkFT，一种面向内存高效的全参数微调框架，其核心是围绕动态激活的工作集重构微调过程。ChunkFT可在不修改网络结构的前提下，对任意子张量进行梯度计算，为任意子网络优化提供了算法基础，同时避免了传统密集梯度计算的开销。我们在确定性设置下提供了ChunkFT的理论收敛性分析。实验表明，使用单张RTX 4090-24GB GPU可对Llama 3-8B进行全参数微调，而两块H800-80GB GPU则支持对Llama 3-70B进行微调。当输入长度为1K时，7B模型微调仅需13.72GB GPU显存。结果表明，ChunkFT在显存占用、运行时间与优化质量方面均表现优异。下游评估显示，其在语言理解、数学推理和MT-Bench任务中持续优于现有内存高效基线，且在多数情况下达到甚至超过全参数微调性能。代码已开源于https://github.com/misonsky/chunk。
原文摘要 · Abstract (English)
This work presents \textsc{ChunkFT}, a memory-efficient fine-tuning framework that reformulates full-parameter fine-tuning around a dynamically activated working set. \textsc{ChunkFT} enables gradient computation for arbitrary sub-tensors without modifying the network architecture, providing an algorithmic foundation for optimizing arbitrary sub-networks while avoiding standard dense gradient computation. We provide a theoretical convergence analysis of \textsc{ChunkFT} in the deterministic setting. Empirically, we apply \textsc{ChunkFT} to fine-tune Llama 3-8B and Llama 3-70B using a single RTX 4090-24GB GPU and 2$\times$ H800-80GB GPUs, respectively. Full-parameter fine-tuning of a 7B model with a 1K input length requires only 13.72GB of GPU memory. The results demonstrate the effectiveness of \textsc{ChunkFT} in memory usage, running time, and optimization quality. Moreover, downstream evaluations on language understanding, mathematical reasoning, and MT-Bench show that \textsc{ChunkFT} consistently outperforms existing memory-efficient baselines. Notably, \textsc{ChunkFT} achieves performance comparable to, and in some cases exceeding, full-parameter fine-tuning. Our repository is on https://github.com/misonsky/chunk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。