arXiv:2604.06829cs.CLcs.AI2026-04

通过挖掘网页链接关系,生成跨文档问答数据,提升大模型预训练知识广度。

WRAP++: Web discoveRy Amplified Pretraining

  • 利用网页超链接发现跨文档关系,合成需跨文档推理的问答对。
  • 将84亿原始文本扩展为800亿跨文档问答数据,规模提升近10倍。
  • 适合追求知识广度与模型可扩展性的大模型训练研究者。

合成数据重述已成为增强大语言模型预训练中知识获取的有效方法。然而,现有方法仅在单文档层面操作,孤立地重写网页内容,导致合成样本局限于文档内部知识,忽略跨文档关联,使事实缺乏充分语境。本文提出WRAP++(Web discovery Amplified Pretraining),通过挖掘网页超链接中的跨文档关系,发现高置信度的关联模式(如双向链接、共提及),并针对每对文档合成需跨文档推理的问答对,从而生成仅靠单个文档无法获得的关系型知识。该方法通过组合式实体对数量的指数增长,实现数据规模的显著放大。在维基百科上实例化后,将约84亿原始文本扩展为800亿跨文档问答数据。在SimpleQA基准上,基于OLMo的7B和32B模型使用WRAP++训练后,显著优于单文档方法,并表现出持续的缩放增益,验证了跨文档知识发现与扩增的优势。

原文摘要 · Abstract (English)

Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

大模型预训练知识增强合成数据跨文档推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。