用输入依赖位置编码学习认知地图，实现无监督路径整合。
MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings
- 通过输入相关矩阵更新位置编码，分离结构关系与具体内容。
- 在导航、嵌套层级等任务中实现接近完美的分布外泛化能力。
- 适合研究具身智能、认知建模及强泛化AI系统的设计者。
认知地图是人类和动物内部存储世界实体间抽象关系的模型，赋予其应对新情境的灵活性，而当前人工智能系统仍缺乏这种分布外（OOD）泛化能力。为此，我们提出基于Transformer的新型架构MapFormer，可从观察数据中无监督地学习认知地图并完成路径整合。该模型通过将输入中的结构关系与其具体内容解耦来构建认知地图，这一特性可通过输入依赖的位置编码矩阵实现——这些矩阵由学习到的李代数生成元组合的指数形式构建。我们开发了两种变体：统一绝对与相对位置编码，分别用于模拟情景记忆（EM）与工作记忆（WM）。我们在多个任务上测试了MapFormer，包括门控、二维导航和嵌套层次结构（Dyck语言）。结果表明，MapFormer显著优于现有模型，在标准模型失效的分布外场景下实现了近乎完美的泛化能力。此外，模型具备可扩展性：在自然数据上的评估显示其困惑度优于基线，表明这些原理可推广至大规模真实世界场景。实验基于交换群上的高效并行计算完成，但模型亦可通过顺序路径整合学习非交换认知地图。总体而言，输入依赖矩阵提供了关键的结构偏差，使抽象关系与内容解耦，从而驱动鲁棒的分布外泛化。
原文摘要 · Abstract (English)
A cognitive map is an internal model which encodes the abstract relationships among entities in the world, giving humans and animals the flexibility to adapt to new situations, with a strong out-of-distribution (OOD) generalization that current AI systems still do not possess. To bridge this gap, we introduce $\textit{MapFormers}$, new Transformer-based architectures, which can learn cognitive maps from observational data and perform path-integration without supervision. Cognitive maps are learned in the model by disentangling structural relationships in the inputs from their specific content, a property that can be achieved by updating position encodings with input-dependent matrices, built as exponentials of learned combinations of Lie-algebra generators. We developed two variants of $\textit{MapFormers}$ that unify absolute and relative positional encoding to model episodic (EM) and working memory (WM), respectively. We tested $\textit{MapFormers}$ on several formal tasks targeting distinct cognitive capacities, including gating, 2D navigation and nested hierarchies (Dyck Languages). Our results demonstrate that $\textit{MapFormers}$ significantly outperform current AI architectures, achieving near-perfect OOD generalization where standard models fail. Furthermore, we show that $\textit{MapFormers}$ are scalable; evaluations on naturalistic data yield perplexity improvements over baselines, suggesting that these principles extend to large-scale, real-world domains. These results are obtained through efficient parallel computation on commutative maps, though our models can also learn non-commutative cognitive maps via sequential path-integration. Overall, these results suggest that input-dependent matrices provide a critical structural bias, by disentangling abstract relations from content in order to drive robust OOD generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。