arXiv:2504.20752cs.CLcs.AI2025-04ICML被引 13

用合成数据提升真实世界多跳推理,让模型从记忆转向真正理解。

Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers

  • 通过合成数据增强知识图谱,提高推理事实占比至可触发突现现象的阈值。
  • 在2WikiMultiHopQA上达到95-100%准确率,显著优于现有基线。
  • 即使合成数据错误也能强化推理能力,适合提升大模型事实推理鲁棒性。

Transformer在众多NLP任务中取得成功,但在真实世界知识稀疏时仍存在明显多步事实推理差距。近期突现现象研究显示,神经网络可在发现底层逻辑模式后从记忆跃迁至完全泛化,但此类研究多基于小规模合成任务。本文首次将突现机制拓展至真实世界事实数据,并通过精心设计的合成数据增强现有知识图谱,使推导事实占比 $ϕ_r$ 超过突现所需阈值。令人惊讶的是,即使合成数据事实错误,也能强化模型的推理回路而非降低精度,因其迫使模型依赖关系结构而非记忆。在多跳推理基准测试中,该方法在2WikiMultiHopQA上实现95-100%准确率,显著优于强基线并达到或超越当前最优结果。我们进一步分析了 $ϕ_r$ 提升如何驱动Transformer内部泛化回路形成。结果表明,基于突现的数据增强可激发模型隐含的多跳推理能力,为大规模语言模型构建更稳健、可解释的事实推理系统开辟新路径。

原文摘要 · Abstract (English)

Transformers have achieved great success in numerous NLP tasks but continue to exhibit notable gaps in multi-step factual reasoning, especially when real-world knowledge is sparse. Recent advances in grokking have demonstrated that neural networks can transition from memorizing to perfectly generalizing once they detect underlying logical patterns - yet these studies have primarily used small, synthetic tasks. In this paper, for the first time, we extend grokking to real-world factual data and address the challenge of dataset sparsity by augmenting existing knowledge graphs with carefully designed synthetic data to raise the ratio $ϕ_r$ of inferred facts to atomic facts above the threshold required for grokking. Surprisingly, we find that even factually incorrect synthetic data can strengthen emergent reasoning circuits rather than degrade accuracy, as it forces the model to rely on relational structure rather than memorization. When evaluated on multi-hop reasoning benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA - substantially improving over strong baselines and matching or exceeding current state-of-the-art results. We further provide an in-depth analysis of how increasing $ϕ_r$ drives the formation of generalizing circuits inside Transformers. Our findings suggest that grokking-based data augmentation can unlock implicit multi-hop reasoning capabilities, opening the door to more robust and interpretable factual reasoning in large-scale language models.

多跳推理突现现象知识图谱合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。