提出模块化架构，分离知识与推理，提升可解释性。
Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
- 用分层变换的交叉注意力显式分离知识存储与推理过程。
- 理论证明标准FFN是该机制的特例，揭示其隐式知识检索作用。
- 适合关注模型可解释性、知识融合的研究者和开发者。
Transformer 在多个领域取得显著成功，但其整体式架构在可解释性、适应性和可扩展性方面存在挑战。本文提出一种新型模块化 Transformer 架构，通过广义交叉注意力机制，将知识与推理显式解耦，并构建全局共享的知识库，配合分层特定变换实现高效知识检索。关键的是，我们提供了严格的数学推导，证明标准 Transformer 中的前馈网络（FFN）是此广义交叉注意力的特例（闭包），揭示了其隐式知识检索的作用，验证了设计合理性。这一理论框架为理解 FFN 提供了新视角，奠定了未来研究的基础，推动模型在外部知识库及其他系统间更丰富的交互，增强可解释性、适应性和可扩展性。
原文摘要 · Abstract (English)
Transformers have achieved remarkable success across diverse domains, but their monolithic architecture presents challenges in interpretability, adaptability, and scalability. This paper introduces a novel modular Transformer architecture that explicitly decouples knowledge and reasoning through a generalized cross-attention mechanism to a globally shared knowledge base with layer-specific transformations, specifically designed for effective knowledge retrieval. Critically, we provide a rigorous mathematical derivation demonstrating that the Feed-Forward Network (FFN) in a standard Transformer is a specialized case (a closure) of this generalized cross-attention, revealing its role in implicit knowledge retrieval and validating our design. This theoretical framework provides a new lens for understanding FFNs and lays the foundation for future research exploring enhanced interpretability, adaptability, and scalability, enabling richer interplay with external knowledge bases and other systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。