用哈密顿动力学分析大模型嵌入空间，发现语义状态具有离散性。
Discrete Semantic States and Hamiltonian Dynamics in LLM Embedding Spaces
- 将大模型嵌入空间类比为量子系统，引入哈密顿形式分析语义关系。
- 证明归一化约束使嵌入空间具备可分析的结构，支持语义跃迁建模。
- 提供新视角理解幻觉机制，适合研究模型可解释性与稳定性者阅读。
我们运用线性代数和哈密顿形式等数学工具，研究大型语言模型（LLM）嵌入空间的结构，借鉴量子力学系统的类比。观察到LLM嵌入表现出显著的离散状态，暗示了离散的语义表示，因此探索这些数学方法在分析语义关系中的应用。我们证明，许多LLM架构中常见的L2归一化约束，使嵌入空间具有适合哈密顿形式分析的结构。推导出余弦相似性与嵌入向量扰动之间的关系，并研究直接与间接的语义转换。此外，从量子启发视角出发，推导出类似零点能量的量，并探讨与Koopman-von Neumann力学的潜在联系。尽管解释需谨慎，但结果表明该方法为深入理解LLM提供了有前景的新途径，或可指导缓解幻觉的新方法。
原文摘要 · Abstract (English)
We investigate the structure of Large Language Model (LLM) embedding spaces using mathematical concepts, particularly linear algebra and the Hamiltonian formalism, drawing inspiration from analogies with quantum mechanical systems. Motivated by the observation that LLM embeddings exhibit distinct states, suggesting discrete semantic representations, we explore the application of these mathematical tools to analyze semantic relationships. We demonstrate that the L2 normalization constraint, a characteristic of many LLM architectures, results in a structured embedding space suitable for analysis using a Hamiltonian formalism. We derive relationships between cosine similarity and perturbations of embedding vectors, and explore direct and indirect semantic transitions. Furthermore, we explore a quantum-inspired perspective, deriving an analogue of zero-point energy and discussing potential connections to Koopman-von Neumann mechanics. While the interpretation warrants careful consideration, our results suggest that this approach offers a promising avenue for gaining deeper insights into LLMs and potentially informing new methods for mitigating hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。