揭示大模型词元空间的几何结构，发现其为分层流形且曲率负向显著。
The structure of the token space for large language models
- 通过维度与里奇曲率估算，分析词元子空间几何结构。
- 三款模型均显示词元空间非普通流形，而是分层结构且曲率显著为负。
- 几何特征与生成流畅性相关，对理解模型行为有重要意义。
大型语言模型通过将语句片段（词元）映射到高维潜在空间来编码自然语言中的相关结构。为深入理解大模型的行为与局限，必须把握该词元子空间的拓扑与几何特性。本文提出词元子空间维度与里奇曲率的估计方法，并应用于三个中等规模开源模型：GPT2、LLEMMA7B 和 MISTRAL7B。结果表明，三者词元子空间均非光滑流形，而是分层流形，各层级上里奇曲率显著为负。此外，维度与曲率与模型生成流畅性呈正相关，暗示这些几何特性对模型行为具有重要影响。
原文摘要 · Abstract (English)
Large language models encode the correlational structure present in natural language by fitting segments of utterances (tokens) into a high dimensional ambient latent space upon which the models then operate. We assert that in order to develop a foundational, first-principles understanding of the behavior and limitations of large language models, it is crucial to understand the topological and geometric structure of this token subspace. In this article, we present estimators for the dimension and Ricci scalar curvature of the token subspace, and apply it to three open source large language models of moderate size: GPT2, LLEMMA7B, and MISTRAL7B. In all three models, using these measurements, we find that the token subspace is not a manifold, but is instead a stratified manifold, where on each of the individual strata, the Ricci curvature is significantly negative. We additionally find that the dimension and curvature correlate with generative fluency of the models, which suggest that these findings have implications for model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。