让大模型训练数据组合与顺序可编程，提升效果并避免人工管理错误。
Mixtera: A Data Plane for Foundation Model Training
- 通过声明式语法定义数据混合比例和访问顺序，无需修改底层存储。
- 支持跨语言、来源等任意属性的动态混合，且不影响训练速度。
- 适合大规模模型训练团队，尤其关注数据策略优化的研究者。
当前最先进的大语言和视觉模型需在万亿级文本上训练，数据来源多样。随着数据集规模扩大，人工管理样本变得耗时且易错。研究表明，数据混合方式及训练中样本访问顺序显著影响模型性能。本文提出 Mixtera，一个用于基础模型训练的数据平面，允许用户以声明式方式指定训练中应使用哪些数据样本、比例及顺序。Mixtera 是部署于现有数据集之上的集中式只读层，独立于文件系统结构，支持跨任意属性（如语言、数据集来源）的混合，并可根据模型反馈动态调整。实验表明，该系统无训练瓶颈，可扩展至 256 张 GH200 超芯片。我们实现了自适应数据优化（ADO）算法并验证其效果，同时探索了混合策略对视觉-语言模型的作用。
原文摘要 · Abstract (English)
State-of-the-art large language and vision models are trained over trillions of tokens that are aggregated from a large variety of sources. As training data collections grow, manually managing the samples becomes time-consuming, tedious, and prone to errors. Yet recent research shows that the data mixture and the order in which samples are visited during training can significantly influence model accuracy. We build and present Mixtera, a data plane for foundation model training that enables users to declaratively express which data samples should be used in which proportion and in which order during training. Mixtera is a centralized, read-only layer that is deployed on top of existing training data collections and can be declaratively queried. It operates independently of the filesystem structure and supports mixtures across arbitrary properties (e.g., language, source dataset) as well as dynamic adjustment of the mixture based on model feedback. We experimentally evaluate Mixtera and show that our implementation does not bottleneck training and scales to 256 GH200 superchips. We demonstrate how Mixtera supports recent advancements in mixing strategies by implementing the proposed Adaptive Data Optimization (ADO) algorithm in the system and evaluating its performance impact. We also explore the role of mixtures for vision-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。