简化文本能提升小模型表现，对零样本任务有不同影响。
Rethinking the Role of Text Complexity in Language Model Pretraining
- 用大模型简化人类文本，保持内容不变但降低复杂度。
- 小模型在简单文本上表现更稳，零样本任务中简单文本利于语言知识。
- 复杂文本更助于世界知识与实体追踪类任务，适合特定目标。
提升预训练数据质量和规模已被证实可增强下游性能，但文本复杂度（阅读难度）的影响仍不明确。本文通过大语言模型简化人类写作的文本，在保持核心内容大致不变的前提下降低表面复杂度（如缩短句子、使用简单词汇和结构），并研究：(i) 复杂度如何影响不同规模模型的语言建模能力？(ii) 是否仅从简单文本中也能学习到有用表示？(iii) 预训练文本复杂度如何影响下游语言理解？我们使用原始与简化数据，从头预训练因果模型（28M-500M参数），并在微调和零样本设置下评估。结果发现，困惑度对模型容量与文本复杂度的交互敏感——小模型在简单文本上退化较少；而复杂度对微调评估影响较小，零样本结果显示：简单文本更利于语言知识任务，复杂文本则更有利于需世界知识和实体追踪的任务。研究揭示不同类型的数据多样性对迁移与零样本性能影响不同，为数据筛选提供了针对性指导。
原文摘要 · Abstract (English)
Improving pretraining data quality and size is known to boost downstream performance, but the role of text complexity--how hard a text is to read--remains less explored. We reduce surface-level complexity (shorter sentences, simpler words, simpler structure) while keeping core content approximately constant and ask: (i) How does complexity affect language modeling across model sizes? (ii) Can useful representations be learned from simpler text alone? (iii) How does pretraining text complexity influence downstream language understanding? We simplify human-written texts using a large language model, pretrain causal models (28M-500M) from scratch on original vs. simplified data, and evaluate them in fine-tuning and zero-shot setups. We find that perplexity is sensitive to the interaction between model capacity and text complexity--smaller models degrade far less on simpler texts--while text complexity has little impact on fine-tuning evaluations, with zero-shot evaluations indicating that simpler texts benefit performance on linguistic knowledge tasks, whereas more complex texts favor tasks requiring world knowledge and entity tracking. Our findings suggest that different types of data diversity affect transfer and zero-shot performance differently, providing insight into tailoring data curation to specific goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。