用物理能量模型优化大模型推理，提升数学解题准确率
Reasoning as Attractor Dynamics: Latent Memory Retrieval via Gibbs-Weighted Energy Minimization
- 将推理过程视为能量最小化动态，通过吉布斯加权采样路径
- 在GSM8K上使Phi-3.5准确率从84.7%提升至90.1%，增益5.38%
- 适合关注推理机制、模型内在动力学的研究者
大型语言模型传统上被视为自回归生成器，但从集体计算视角看，它们是高维稠密关联记忆，以潜在吸引子形式存储复杂推理模式。本文研究数学推理的能量景观，认为正确推理链对应模型输出分布中深而宽的吸引子盆地（平坦极小值），而幻觉则表现为尖锐且不稳定的局部极小值。为利用该几何结构，我们提出一种基于轨迹谱熵吉布斯测度的检索机制。通过采样多个推理路径并按其逆能量加权（$P /propto e^{-βE}$），近似关联记忆的平衡分布，有效实现系统向稳健解的‘松弛’。实验证明，该受物理启发的机制使微软Phi-3.5在GSM8K上的表现提升5.38%（84.7% → 90.1%），表明推理更应被建模为进入吸引子盆地的动态收敛过程，而非贪婪的逐词预测。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are traditionally viewed as autoregressive generators. However, from the perspective of collective computation, they function as high-dimensional Dense Associative Memories that store complex reasoning patterns as latent attractors. In this work, we investigate the energy landscape of mathematical reasoning. We posit that correct reasoning chains correspond to deep, wide attractor basins ("flat minima") in the model's output distribution, whereas hallucinations manifest as sharp, unstable local minima. To exploit this geometry, we introduce a retrieval mechanism based on a Gibbs measure of the trajectory's spectral entropy. By sampling multiple reasoning paths and weighting them by their inverse energy ($P \propto e^{-βE}$), we approximate the equilibrium distribution of the associative memory, effectively ``relaxing'' the system into a robust solution. Empirically, this physics-inspired mechanism improves Microsoft Phi-3.5 performance on GSM8K by 5.38\% (84.7\% $\to$ 90.1\%), demonstrating that inference is better modeled as a dynamic settling process into an attractor basin rather than greedy next-token prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。