135M小模型在单张GPU上用130亿数据训练，性能达同类87%。
L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
- 单卡运行135M模型，仅用130亿文本训练
- 在六任务评测中平均得分0.415，为基准模型的87%
- 公开可审计过程，适合研究小模型高效训练
小语言模型成本低、可在本地部署，但主流135M级系统通常需数百亿至数万亿标记在大型集群上训练。本文研究极端资源受限场景：完整134.5M参数语言模型流程在单张NVIDIA L20 GPU上完成。释放的L20-Edu-135M检查点使用约130亿预训练标记：100亿FineWeb-Edu标记，外加30亿教育、数学、代码与推理混合数据。文档涵盖架构设计、数据筛选机制、跨源MinHash/LSH近似去重、段落去重、基准重叠剔除、吞吐优化、监督微调（含权重插值）及基于可验证奖励的强化学习（RLVR）在GSM8K上的应用。自运行零样本六任务评估中，模型平均得分为0.4150，低于SmolLM-135M（0.4767）和SmolLM2-135M（0.4917），但其得分达到后两者的87.1%，而训练标记量仅为后两者2.17%。该比例具描述性，非统计等效或受控缩放律证据。模型优于多个旧版100M–160M公共基线。直接采用GRPO风格的RLVR使GSM8K精确匹配率从1.82%降至1.59%（192令牌生成）和1.21%（320令牌生成）。单次运行结果揭示具体失效模式，而非对RLVR的普遍下限判断。贡献在于提供可审计的资源受限案例研究，非前沿性能宣称。
原文摘要 · Abstract (English)
Small language models are cheap to serve and feasible on local hardware, but strong public 135M-class systems are commonly trained with hundreds of billions to trillions of tokens on large clusters. We study a sharply resource-constrained regime: a complete 134.5M-parameter language-model pipeline executed on one NVIDIA L20 GPU. The released checkpoint, L20-Edu-135M, receives approximately 13B pretraining tokens: 10B FineWeb-Edu tokens followed by a 3B-token educational, mathematics, code, and reasoning mixture. We document the architecture, data gates, cross-source MinHash/LSH near-deduplication, segment deduplication, benchmark-overlap removal, throughput optimization, supervised fine-tuning (SFT) with weight interpolation, and reinforcement learning from verifiable rewards (RLVR) on GSM8K. In a self-run zero-shot six-task harness, L20-Edu-135M obtains a mean score of 0.4150. It trails SmolLM-135M (0.4767) and SmolLM2-135M (0.4917), but its mean is 87.1% of SmolLM-135M's while its nominal token count is 2.17% as large. This ratio is descriptive, not evidence of statistical equivalence or a controlled scaling law. The model exceeds several older 100M-160M public baselines under the same harness. Direct GRPO-style RLVR decreases GSM8K exact-match accuracy from 1.82% to 1.59% (192-token completions) and 1.21% (320-token completions). These single-run results identify a concrete failure mode rather than establishing a general lower bound on RLVR. The contribution is an auditable resource-constrained case study, not a state-of-the-art claim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。