arXiv:2607.28109cs.AI2026-07被引 1

通过构建有结构的教材级数据,显著提升语言模型训练效果。

Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

论文配图:Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
图 1 · 摘自论文原文
  • 将文本按主题聚类并规划目录,组装成完整教材。
  • 相比独立段落,整体组织使下游任务平均提升1.09分。
  • 适合关注合成数据设计与模型预训练的研究者。

合成教材数据已提升语言模型预训练效果,但以往研究多将其归因于生成内容或局部重写风格。本文探讨另一因素:相关内容是否被组织为连贯的教材级文档。我们提出可扩展的合成流水线,从预训练语料中检索材料,聚类为主题单元,规划层级目录,并基于源文本组装成完整书籍(全量设置),生成686,000本教材(320亿词元),覆盖15,000多个学科。在中期训练中用该数据替换自然教材,下游性能平均提升+1.09。受控对比显示:内容匹配的拆分条件(每节独立)带来+1.02增益,说明文档组织是关键;长度匹配的随机拼接仍低于全量设置,排除长度影响;检索池匹配的重写条件(无聚类、目录规划和组装)仅得+1.17增益,证明结构化合成的价值。在Llama3-8B上,全量设置仍优于随机拼接与自然教材,支持教材级组织是合成预训练数据设计的重要维度。

原文摘要 · Abstract (English)

Synthetic textbook data has improved language model pre-training, but prior work largely treats the benefit as a property of generated content or local rewriting style. We study a different factor: whether related content is organized into coherent book-level documents. We contribute both a scalable synthesis pipeline and controlled evidence that this organization matters. The pipeline retrieves source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books (our Full setting), yielding 686K textbooks (32B tokens) across 15,000+ disciplines. Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average. Controlled comparisons then disentangle the relevant design factors. A content-matched Split condition holds generated text and tokens fixed but treats each section as an independent document; Full's +1.02 mean gain isolates document packaging. A length-matched RandomConcat control that joins sections from different books remains below Full, ruling out document length alone. A retrieval-pool-matched Rephrase condition independently rewrites individual retrieved documents under the same audience-by-style scheme, without clustering, TOC planning, or book assembly; Full's +1.17 gain demonstrates the value of structured synthesis. On Llama3-8B, Full likewise outperforms both RandomConcat and Natural Books, supporting book-level organization as a useful axis for synthetic pre-training data design.

合成数据预训练结构化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。