构建具空间结构的语言模型，模拟大脑语言区的组织规律。
TopoLM: brain-like spatio-functional organization in a topographic language model
- 在模型中引入二维空间表示，通过空间平滑损失约束表征分布。
- 模型生成的聚类与大脑语言区功能分区高度匹配，涵盖语义与句法特征。
- 适合研究脑科学与语言模型对齐机制的科研人员参考。
大脑神经元在空间上存在组织性，邻近区域常具有相似响应特性。人类语言系统中，实验观察到句法和语义类别形成集群，但其内在机制仍不明确。本文受视觉领域工作启发，提出TopoLM——一种具备显式二维空间表示的Transformer语言模型。通过结合下一词预测目标与空间平滑损失，模型表征自发形成可解释的语义聚类，且与人脑语言系统功能组织高度一致。该模型成功预测了皮层语言系统的空间-功能组织模式，并复现了人脑皮层中针对细粒度语言特征的选择性功能簇。结果表明，人类语言系统的功能组织由统一的空间优化目标驱动，为脑内语言处理提供了兼具功能与空间对齐的建模范式。
原文摘要 · Abstract (English)
Neurons in the brain are spatially organized such that neighbors on tissue often exhibit similar response profiles. In the human language system, experimental studies have observed clusters for syntactic and semantic categories, but the mechanisms underlying this functional organization remain unclear. Here, building on work from the vision literature, we develop TopoLM, a transformer language model with an explicit two-dimensional spatial representation of model units. By combining a next-token prediction objective with a spatial smoothness loss, representations in this model assemble into clusters that correspond to semantically interpretable groupings of text and closely match the functional organization in the brain's language system. TopoLM successfully predicts the emergence of the spatio-functional organization of a cortical language system as well as the organization of functional clusters selective for fine-grained linguistic features empirically observed in human cortex. Our results suggest that the functional organization of the human language system is driven by a unified spatial objective, and provide a functionally and spatially aligned model of language processing in the brain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。