arXiv:2609.07821cs.CLcs.AI2026-09

用几何动态优化大模型推理,让关键步骤显式保留,冗余步骤压缩成隐向量。

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM

论文配图:A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
图 1 · 摘自论文原文
  • 将推理过程建模为三维空间中的轨迹,按方向一致性决定保留或压缩步骤。
  • 在多个基准上提升准确率最高2.6%,响应长度减半,计算效率提高2.29倍。
  • 适合追求高效推理的部署场景,尤其对资源受限环境友好。

链式思维(CoT)虽提升大语言模型(LLM)推理能力,但带来巨大计算与上下文开销。现有方法或通过硬性剪枝丢失中间信息,或缺乏连续压缩的合理准则。本文提出A*-Thought-V2,一种基于大模型几何动态的推理框架,将CoT建模为隐藏状态轨迹,以显式-隐式交替的潜在架构替代硬删除。通过将问题、步骤与解表示投影至3D PCA空间,测量每一步局部转移与全局问题到解方向的一致性:一致步骤保持显式文本,偏离步骤则压缩为连续潜在标记。方向角同时捕捉局部语义与推理动态:小角度对应直接执行与答案生成,大角度常涉及校验、修正与分支探索;其时间变化揭示探索、收敛与精炼阶段。为训练该架构,引入逐步嵌入强制机制,将冗余步骤聚合成单一潜在嵌入,并采用软多模态词表分布监督潜在标记,而非硬性one-hot标签。在Qwen3.5-9B和Qwen3.6-27B上,跨六项领域内与领域外基准测试表明,A*-Thought-V2平均准确率提升最高达2.6%,响应长度减少一半,单位计算量准确率提升2.29倍,预处理与训练时间分别降低94.6%与最高80.3%。表征分析显示,潜在状态形成与文本状态分离的紧凑区域,潜在标记位置熵更高,反映更广软目标,促进更丰富的步骤级特征学习。

原文摘要 · Abstract (English)

Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation and context costs. Existing methods either lose intermediate information through hard pruning or lack a principled criterion for continuous compression. We present A*-Thought-V2, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture. After projecting question, step, and solution representations into a 3D PCA space, it measures alignment between each local transition and global question-to-solution direction. Aligned steps remain explicit text, whereas deviating steps are compressed into continuous latent tokens. Directional angles capture both local semantics and reasoning dynamics: small angles indicate direct execution and answer formation, while large angles more frequently involve checking, correction, and branch exploration; their temporal variation reveals exploration, convergence, and refinement stages. To train this architecture, we introduce stepwise embedding forcing, which pools each redundant step into a single latent embedding, and label forcing, which supervises that latent token with a soft multi-modal vocabulary distribution instead of a hard one-hot label. Experiments on Qwen3.5-9B and Qwen3.6-27B across six in-domain and out-of-domain benchmarks show that A*-Thought-V2 improves average accuracy by up to 2.6% while reducing response length by up to half, increasing Accuracy per Computation Unit by 2.29$\times$, and reducing preprocessing and training time by 94.6% and up to 80.3%, respectively. Representation analyses suggest that latent states form a compact region distinct from textual states, while higher entropy at latent-token positions reflects broader soft targets that encourage richer step-level feature learning.

大模型推理链式思维高效压缩潜在表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。