用多样性采样训练模型，能大幅减少数据量且效果更好。
A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
- 用多样性算法挑选训练数据，替代随机采样。
- 150M token多样数据训练483小时，性能媲美2.4B token随机数据训练1775小时。
- 在部分任务上性能提升10分，适合资源有限的研究者。
近年来，多样性在自然语言处理领域受到关注。当前最先进的Transformer模型如ModernBERT通常使用超大规模预训练数据集，但这些数据集更注重规模而非多样性。本研究旨在探究多样性对ModernBERT预训练的影响，目标是减少预训练数据量，同时保持相当的性能。我们比较了多种基于多样性的采样算法，发现某些任务中，使用多样性采样相比同规模随机采样的数据可提升10个百分点。此外，一个在150M tokens多样性数据上训练483小时的模型，其性能可与在2.4B tokens随机数据上训练1775小时的模型相当。
原文摘要 · Abstract (English)
Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons for an investigation of the impact of diversity on the ModernBERT pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, so as to pick the best one. We find that diversity-driven sampling allows in some tasks to gain 10 points relative to randomly-sampled pre-training data of commensurate size. We also see that a model pre-trained for 483h on a diversity-driven dataset of 150M tokens can yield a commensurate performance to a model pre-trained for 1,775h on a randomly-driven dataset of 2.4B tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。