用细粒度词元特征采样数据，让模型在小数据下仍能高效训练。
Target-Aware Language Modeling via Granular Data Sampling
- 基于多粒度词元的重要性采样，平衡压缩与表征能力
- 仅用约1%数据，性能媲美全量数据集，优于随机采样
- 适合需要领域优化但不想牺牲通用性的场景
语言模型预训练通常面向广泛用途，涵盖多样化数据源。但在某些情况下，我们希望模型在特定领域表现优异，同时不显著影响其他任务性能。一种低成本且直接的方法是使用低维数据特征进行采样，从而筛选大规模预训练数据以适配特定领域。本文重新审视基于包含多粒度词元的n-gram特征的重要性采样方法，该方法在句子压缩与表征能力间取得良好平衡。我们观察到所采样数据与下游目标任务性能高度相关，同时保持在其他任务上的有效性。由此提出一种新的数据采样范式，使语言模型可在选定文档上更高效地进行预训练。在八个基准测试中，仅使用约1%的数据，预训练模型的表现即可媲美完整RefinedWeb数据集，且优于随机采样样本，模型规模从125M到1.5B均适用。
原文摘要 · Abstract (English)
Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with $\sim$1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。