通过递归隐空间推理，提升Transformer在分布外任务中的泛化能力。
Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning
- 设计四类架构机制增强模型对未知分布的推理能力。
- 在模运算任务上实现显著优于基线的分布外泛化性能。
- 适合关注大模型推理能力与可解释性的研究者。
系统性、组合式的分布外（OOD）泛化仍是机器学习的核心挑战，也是现代语言模型涌现推理能力的关键瓶颈。本文以GSM8K风格的计算图模运算任务为测试平台，研究Transformer网络的分布外泛化能力。提出并探索四种架构机制：(i) 输入自适应递归；(ii) 算法监督；(iii) 通过离散瓶颈锚定隐表示；(iv) 显式纠错机制。这些机制共同构成一种原生且可扩展的Transformer隐空间推理方法，具备稳健的算法泛化能力。我们结合详细的机制可解释性分析，揭示这些机制如何催生稳健的分布外泛化能力。
原文摘要 · Abstract (English)
Systematic, compositional generalization beyond the training distribution remains a core challenge in machine learning -- and a critical bottleneck for the emergent reasoning abilities of modern language models. This work investigates out-of-distribution (OOD) generalization in Transformer networks using a GSM8K-style modular arithmetic on computational graphs task as a testbed. We introduce and explore a set of four architectural mechanisms aimed at enhancing OOD generalization: (i) input-adaptive recurrence; (ii) algorithmic supervision; (iii) anchored latent representations via a discrete bottleneck; and (iv) an explicit error-correction mechanism. Collectively, these mechanisms yield an architectural approach for native and scalable latent space reasoning in Transformer networks with robust algorithmic generalization capabilities. We complement these empirical results with a detailed mechanistic interpretability analysis that reveals how these mechanisms give rise to robust OOD generalization abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。