对比不同数据筛选策略对语言模型性能的影响。
Investigating the Impact of Data Selection Strategies on Language Model Performance
- 比较随机选择与分布对齐的数据筛选方法
- 发现n-gram特征能更好匹配目标分布
- 嵌入特征可提供互补优势，适合数据优化研究者
数据筛选对提升语言模型性能至关重要，尤其在训练数据分布与目标分布对齐时。本研究探讨不同数据筛选方法和特征类型对模型表现的影响。通过对比基线随机选择与分布对齐方法的实验，评估数据子集选择是否影响下游任务性能，n-gram特征能否增强与目标分布的对齐，以及基于嵌入的神经特征是否带来额外收益。结果揭示了数据筛选策略与模型训练效率之间的相互作用机制。所有代码可在GitHub仓库获取。
原文摘要 · Abstract (English)
Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。