通过几何视角揭示大模型如何逐步提炼决策信息
Geometry of Decision Making in Language Models
- 用内在维度分析模型各层隐藏表征的几何结构
- 发现中间层扩展空间、后期压缩至低维决策空间
- 为理解模型泛化与推理提供新几何解释，适合研究者参考
大语言模型在多样化任务中表现出强泛化能力，但其内部决策机制仍不清晰。本文通过内在维度（ID）视角，研究28个开源Transformer模型在多项选择题问答（MCQA）任务中的隐藏表征几何特性。我们使用多种估计器分析各层ID，并量化每层在MCQA任务上的表现。结果表明：早期层处于低维流形，中间层拓展该空间，后期再次压缩，最终收敛至与决策相关的低维表示。这说明大模型隐式地将语言输入投影到与任务相关、结构化的低维流形上，为理解语言模型中泛化与推理的产生提供了新的几何洞察。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show strong generalization across diverse tasks, yet the internal decision-making processes behind their predictions remain opaque. In this work, we study the geometry of hidden representations in LLMs through the lens of \textit{intrinsic dimension} (ID), focusing specifically on decision-making dynamics in a multiple-choice question answering (MCQA) setting. We perform a large-scale study, with 28 open-weight transformer models and estimate ID across layers using multiple estimators, while also quantifying per-layer performance on MCQA tasks. Our findings reveal a consistent ID pattern across models: early layers operate on low-dimensional manifolds, middle layers expand this space, and later layers compress it again, converging to decision-relevant representations. Together, these results suggest LLMs implicitly learn to project linguistic inputs onto structured, low-dimensional manifolds aligned with task-specific decisions, providing new geometric insights into how generalization and reasoning emerge in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。