Chronicals框架让大模型微调提速3.5倍，突破显存瓶颈。
Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth
- 融合Triton内核减少75%内存传输，加速RMSNorm、SwiGLU等操作。
- 在线softmax计算将日志内存从5GB降至135MB，大幅降低显存占用。
- 适合追求极致训练效率的开发者，尤其在40GB显存设备上表现优异。
大语言模型微调受限于显存：一个70亿参数模型需84GB显存——其中14GB用于权重，14GB用于梯度，56GB用于FP32优化器状态，超过A100-40GB显存容量。我们提出Chronicals，一个开源训练框架，在多项协同优化下实现比Unsloth快3.51倍的性能提升。其核心技术包括：(1) 融合Triton内核，通过RMSNorm（7倍）、SwiGLU（5倍）和QK-RoPE（2.3倍）融合，消除75%内存流量；(2) Cut Cross-Entropy在线计算软最大值，将日志内存从5GB降至135MB；(3) LoRA+使用理论推导的16倍差异学习率，优化适配矩阵更新；(4) Best-Fit Decreasing序列打包，恢复因填充浪费的60%-75%算力。在A100-40GB上，Qwen2.5-0.5B全量微调达到41,184 tokens/秒，优于Unsloth的11,736 tokens/秒（3.51倍）；LoRA rank=32时达11,699 tokens/秒，是Unsloth MAX的4.10倍。值得注意的是，原Unsloth报告的46,000 tokens/秒实为零梯度范数，模型未真实训练。本文提供完整数学基础：在线softmax正确性证明、FlashAttention I/O复杂度界O(N²d²M⁻¹)、LoRA+学习率推导及装箱近似保证。所有代码、基准测试与证明已公开于https://github.com/Ajwebdevs/Chronicals，支持pip安装：https://pypi.org/project/chronicals/。
原文摘要 · Abstract (English)
Large language model fine-tuning is bottlenecked by memory: a 7B parameter model requires 84GB--14GB for weights, 14GB for gradients, and 56GB for FP32 optimizer states--exceeding even A100-40GB capacity. We present Chronicals, an open-source training framework achieving 3.51x speedup over Unsloth through four synergistic optimizations: (1) fused Triton kernels eliminating 75% of memory traffic via RMSNorm (7x), SwiGLU (5x), and QK-RoPE (2.3x) fusion; (2) Cut Cross-Entropy reducing logit memory from 5GB to 135MB through online softmax computation; (3) LoRA+ with theoretically-derived 16x differential learning rates between adapter matrices; and (4) Best-Fit Decreasing sequence packing recovering 60-75% of compute wasted on padding. On Qwen2.5-0.5B with A100-40GB, Chronicals achieves 41,184 tokens/second for full fine-tuning versus Unsloth's 11,736 tokens/second (3.51x). For LoRA at rank 32, we reach 11,699 tokens/second versus Unsloth MAX's 2,857 tokens/second (4.10x). Critically, we discovered that Unsloth's reported 46,000 tokens/second benchmark exhibited zero gradient norms--the model was not training. We provide complete mathematical foundations: online softmax correctness proofs, FlashAttention IO complexity bounds O(N^2 d^2 M^{-1}), LoRA+ learning rate derivations from gradient magnitude analysis, and bin-packing approximation guarantees. All implementations, benchmarks, and proofs are available at https://github.com/Ajwebdevs/Chronicals with pip installation via https://pypi.org/project/chronicals/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。