arXiv:2505.20340cs.CLcs.AI2025-05

揭示大模型生成时潜在空间的动态演化规律,提出可验证的轨迹建模框架。

Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validation

  • 将语言模型生成视为低维语义流形上的受控动力系统演化过程。
  • 三个可验证指标连续预测文本质量,效果在1080次实验中稳定显著。
  • 可直接用于优化生成质量,实测困惑度降低至14.6,适合模型可控生成研究者。

理解生成过程中潜在表示的演化是大语言模型可解释性中的核心开放问题。本文提出 extbf{动力流形演化理论}(DMET),将LLM生成建模为在低维语义流形上沿轨迹演化的受控动力系统。DMET将Transformer组件与由语义势能$V$驱动的一阶常微分方程建立结构对应关系,并通过三个可证伪的代理指标刻画轨迹几何:状态连续性$C$、吸引子聚类质量$Q$和拓扑持久性$P$,分别针对局部平滑性、中尺度吸引子结构和全局拓扑组织。在六种模型架构、四种任务类型及1,080次实验中,三者均在控制解码参数后一致预测文本质量——对数困惑度、语法正确性和跨句连贯性,关联性在Benjamini--Hochberg校正下仍显著。消融与合理性检验确认效应源于真实轨迹结构而非静态分布伪影。此外,对$C$的在线监测驱动自适应解码控制器,使困惑度从48.5降至14.6,证明潜在动态特性可直接转化为可操作的生成控制。

原文摘要 · Abstract (English)

Understanding how latent representations evolve during generation is a central open problem in large language model interpretability. We introduce \textbf{Dynamical Manifold Evolution Theory} (DMET), a phenomenological framework that models LLM generation as a controlled dynamical system evolving along a trajectory on a low-dimensional semantic manifold. DMET formalizes the structural correspondence between Transformer components and a first-order ODE governed by a semantic potential $V$, and characterizes trajectory geometry through three falsifiable proxy metrics: state continuity $C$, attractor clustering quality $Q$, and topological persistence $P$, targeting local smoothness, meso-scale basin structure, and global topological organization, respectively. Across six model architectures, four task types, and 1,080 experimental runs, all three metrics consistently predict text quality outcomes -- log-perplexity, grammaticality, and cross-sentence coherence -- after controlling for decoding parameters, with associations surviving Benjamini--Hochberg correction. Ablation and sanity-check experiments confirm that the effects arise from genuine trajectory structure rather than static distributional artefacts. Furthermore, online monitoring of $C$ drives an adaptive decoding controller that reduces perplexity from 48.5 to 14.6 relative to a fixed-parameter baseline, demonstrating that latent dynamics characterization translates directly into actionable generation control.

大模型可解释性潜在空间演化生成控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。