开源小模型训练框架,代码可读性强,适合教学与研究
OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

- 模型结构用模块化代码直观表达,像看图布线一样清晰
- 348M参数模型在4张显卡上实现90.6%弱扩展效率
- 支持从教学笔记本直接跑完整训练,适合学生和研究员
OpenLanguageModel(OLM)是一个开源的PyTorch库,用于构建和预训练小型语言模型,同时保持其内部机制透明可见。在OLM中,模型代码如同架构图:组件为普通模块,Block、Residual、Repeat和Parallel描述它们的连接方式。由此生成的模型可无修改地从教学笔记本直接进入完整预训练或研究消融实验。OLM将这种可读模型层与分词器、本地及流式数据集、优化器、混合精度、回调函数、检查点以及支持CPU、单GPU和单节点多GPU的硬件感知执行无缝集成。我们通过追踪GPT-2从图到代码,启动FineWeb-Edu训练脚本,替换一个注意力组件,并让AutoTrainer自动配置可用设备,完整演示了该流程。该包包含9种常见模型族的27个预设,文档从语言模型基础逐步深入至架构研究。验证显示与独立参考实现高度一致,348M参数任务在4张显卡上达到90.6%弱扩展效率,架构修改紧凑,早期可用性反馈积极。OLM采用MIT许可证,可通过PyPI、GitHub和官网获取。
原文摘要 · Abstract (English)
OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。