发现大模型推理本质是受几何与信息约束的动态过程。
Reasoning emerges from constrained inference manifolds in large language models
- 从内部表征演化中发现推理自组织为低维流形。
- 有效推理需满足表达力、自发压缩与信息保真三条件。
- 提出无需标签的诊断方法，可评估模型内在推理质量。
大型语言模型中的推理主要通过标注基准评估，混淆了任务表现与内部推断质量。本文将推理视为内在动力学过程，研究推断期间内部表征的演化。发现推断动态在高维表征空间中持续自组织为低维流形。尽管这种几何压缩普遍存在，但不足以保证稳定可靠的推理。有效推理动态仅在特定结构约束下出现，该约束包含三个条件：充分的表征表达能力、自发的流形压缩，以及压缩子空间内非退化的信息体积保持。偏离此区间的模型表现出典型病态推断动态。基于这些发现，我们提出一种仅依赖内部动态计算的统一、无标签诊断方法。结果表明，大模型推理本质上由几何与信息约束所主导，为基准中心评估提供了互补框架。
原文摘要 · Abstract (English)
Reasoning in large language models is predominantly evaluated through labeled benchmarks, conflating task performance with the quality of internal inference. Here we study reasoning as an intrinsic dynamical process by examining the evolution of internal representations during inference. We find that inference-time dynamics consistently self-organize into low-dimensional manifolds embedded within high-dimensional representation spaces. we find that such geometric compression, although pervasive, is not sufficient for stable or reliable reasoning. Instead, effective reasoning dynamics emerge within a constrained structural regime characterized by three conditions: adequate representational expressivity, spontaneous manifold compression, and preservation of non-degenerate information volume within the compressed subspace. Models outside this regime exhibit characteristic pathological inference dynamics. Based on these insights, we introduce a unified, label-free diagnostic computed solely from internal dynamics. These findings suggest that reasoning in LLMs is fundamentally governed by geometric and informational constraints, offering a complementary framework to benchmark-centric assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。