用合成问答与文档混合训练，让模型突破RAG性能天花板
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
- 混合使用合成问答和文档，互补提升学习信号
- 在QuaLITY上相对RAG提升4.4%，五组测试中四次胜出
- 适合想超越RAG的模型训练者，尤其关注长文本理解
合成数据增强有助于语言模型在数据稀缺领域学习新知识。然而，单纯增加合成数据量或使用更强生成器，其效果提升逐渐减弱，难以超越RAG。为打破这一瓶颈，我们提出合成混合训练（Synthetic Mixed Training），结合合成问答对与合成文档，利用二者互补的训练信号，实现合成数据量与生成器强度双增长下的对数线性提升。该方法使模型在长文档阅读理解基准QuaLITY上相对RAG提升2.6%。此外，我们引入聚焦重写（Focal Rewriting）技术，显式以特定问题为条件生成文档，提升合成文档多样性，获得更陡峭的对数线性增长曲线。最终方案在训练Llama 8B模型时，相对RAG提升4.4%。在多个模型与基准（QuaLITY、LongHealth、FinanceBench）下，六组中有五组超越RAG，平均提升2.6%，与RAG结合后最高达9.1%。
原文摘要 · Abstract (English)
Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synthetic tokens or using stronger generators yields diminishing returns below the performance of RAG. To break the RAG ceiling, we introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents. This leverages their complementary training signals, and enables log-linear improvements as both synthetic data volume and generator strength increase. This allows the model to outperform RAG by a 2.6% relative gain on QuaLITY, a long-document reading comprehension benchmark. In addition, we introduce Focal Rewriting, a simple technique for synthetic document generation that explicitly conditions document generation on specific questions, improving the diversity of synthetic documents and yielding a steeper log-linear scaling curve. On QuaLITY, our final recipe trains a Llama 8B model that outperforms RAG by 4.4% relatively. Across models and benchmarks (QuaLITY, LongHealth, FinanceBench), our training enables models to beat RAG in five of six settings, outperforms by 2.6%, and achieves a 9.1% gain when combined with RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。