用合成数据预预训练，让大模型更抗噪声数据干扰。
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
- 用带可学习时序结构的合成数据做轻量预预训练。
- 10亿参数模型用6500万合成词元，省49%真实语料仍达同等效果。
- 适合处理高噪声预训练数据的场景，提升模型鲁棒性。
大语言模型依赖海量网络语料进行预训练，但数据中固有的噪声会掩盖有效模式，降低模型性能。尽管数据清洗能缓解问题，实际预训练语料仍存在噪声。本文研究基于合成数据与可学习时序结构的轻量级预预训练（PPT）是否有助于提升预训练阶段对噪声的鲁棒性。在多种污染设置下，该方法均显著增强模型抗噪能力，且噪声越高，增益越明显。对于10亿参数模型，仅使用6500万合成词元的PPT阶段，即可在不同噪声水平下实现与基线相当的最终损失，同时减少最多49%的真实文本预训练词元。机制分析表明，PPT不会立即抑制对噪声词元的关注，而是使初始化模型在噪声预训练过程中逐步降低对受损词元的关注度。这说明合成预预训练抑制了噪声自建模，并引导后续优化轨迹。代码已开源：https://github.com/guox18/formal-language-prepretraining。
原文摘要 · Abstract (English)
Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49\% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。