在16MB内存限制下，优化语言模型性能的实战挑战
Parameter Golf: What Really Works?
- 通过2037个提交分析84种优化技术，量化其对模型压缩效果的影响
- 最终模型在10分钟内完成训练，比特每字节降至1.058，提升13.6%
- 揭示少数方法在多层优化中仍有效，适合追求极致部署效率的研究者
参数高尔夫（Parameter Golf）以开放社区挑战形式提出问题：在严格资源预算下，语言模型能提升到何种程度？参赛者需在8块H100 SXM GPU上，于10分钟内完成训练，并将完整模型（训练代码+压缩权重）控制在16 MB以内。模型质量以比特每字节（BPB）衡量，即编码未见文本每个字节所需的平均比特数。我们分析了2,037个拉取请求和1,430个干净评分提交，构建了84种优化技术的分类体系，并量化每项技术对BPB的贡献。经过三个阶段，经验证的排行榜得分从1.2244下降至1.058 BPB，降幅达13.6%，尽管单个技术通常仅带来小于1%的改善。研究发现，多数优化技术在竞争性提交中效果衰减，仅少数方法在多种模型堆栈中持续有效。
原文摘要 · Abstract (English)
How far can a language model improve under a strict artifact budget? Parameter Golf posed this question as an open community challenge in which participants trained the best language model, with the complete artifact (training code + compressed weights) required to fit within 16 MB and be trained in under ten minutes on 8xH100 SXM GPUs. Quality was measured in bits-per-byte (BPB), the average number of bits required to encode each byte of unseen text. We analyze 2,037 pull requests and 1,430 clean scored submissions from the contest, build a taxonomy of 84 optimization techniques, and measure each technique's contribution to BPB. The verified leaderboard score dropped from 1.2244 to 1.058 BPB across three phases -- a 13.6% reduction, despite individual techniques rarely improving BPB by more than 1%. We show that most gains in techniques shrink across competitive submissions, isolating the few methods that improve performance across stacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。