Modalities让大模型训练与实验更高效，支持万亿级文本和百亿参数规模的系统性对比研究。
Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research
- 基于PyTorch原生设计，集成先进并行策略，实现高效训练与可复现的系统性消融实验。
- 支持万亿级别文本、百亿参数规模的预训练与消融，显著降低实验开发成本。
- 模块化配置设计，便于科研人员快速复现和扩展新实验，适合大规模模型研究者。
当前的大规模语言模型（预）训练与研究工作通常需投入大量算力进行大规模消融实验。尽管这些实验成本高昂，现有开源框架在工具支持方面仍显不足，常迫使研究人员自行编写封装代码。我们提出Modalities，一个端到端的PyTorch原生框架，从两个层面将数据驱动的大模型研究与大规模训练相结合：首先，通过集成最先进的并行化策略，实现万亿级文本与百亿参数规模下的高效预训练与系统性消融；其次，采用模块化设计与声明式、自包含的配置方式，达到现有框架难以实现的可复现性和可扩展性水平。
原文摘要 · Abstract (English)
Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。