发现Transformer层间存在降维规律，逼近人类语义空间。
Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
- 通过几何分析揭示各层令牌动态演化规律
- 模型在约10维子流形上压缩语义信息
- 适合关注模型可解释性与诊断的读者
大型语言模型（LLMs）的令牌表征几何演化存在根本矛盾：人类语言内在语义空间维度低（约10¹维），而现代LLMs采用高维嵌入（约10³维）并通过Transformer架构处理。本文构建几何框架，追踪多个架构中令牌在层间的动态变化，揭示其先扩展后收缩的模式——令牌先扩散至“工作空间”，再逐步投影到低维子流形。研究发现，工作空间维度与参数敏感性能呈负相关，高效模型倾向于将令牌压缩至约10维子流形，接近人类语义空间。该工作不仅将Transformer层重构为介于高维计算与低维语义间的投影器，提升模型可解释性，还提供无需任务特定评估的模型诊断工具。
原文摘要 · Abstract (English)
The geometric evolution of token representations in large language models (LLMs) presents a fundamental paradox: while human language inherently organizes semantic information in low-dimensional spaces ($\sim 10^1$ dimensions), modern LLMs employ high-dimensional embeddings ($\sim 10^3$ dimensions) processed through Transformer architectures. To resolve this paradox, this work bridges this conceptual gap by developing a geometric framework that tracks token dynamics across Transformers layers. Through layer-wise analysis of intrinsic dimensions across multiple architectures, we reveal an expansion-contraction pattern where tokens diffuse to a "working space" and then progressively project onto lower-dimensional submanifolds. Our finding implies a negative correlation between the working space dimension and parameter-sensitive performance of the LLMs, and indicates that effective models tend to compress tokens into approximately 10-dimensional submanifolds, closely resembling human semantic spaces. This work not only advances LLM interpretability by reframing Transformers layers as projectors that mediate between high-dimensional computation and low-dimensional semantics, but also provides practical tools for model diagnostics that do not rely on task-specific evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。