arXiv:2605.09008cs.LGcs.CL2026-05

提出新方法提升大模型推理时的剪枝效率,缓解长链推理延迟问题。

Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models

论文配图:Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models
图 1 · 摘自论文原文
  • 基于梯度流构建连续动能框架,识别逻辑推理关键路径。
  • 40%稀疏度下在GSM8K上达13.34%准确率,优于现有最优基线。
  • 适合关注大模型推理加速与推理能力保持的研究者。

链式思维(CoT)提示显著提升了大语言模型的推理能力,但测试时计算量增大导致长序列生成,引发严重推理延迟和键值缓存瓶颈。结构化剪枝虽能缓解静态参数负担,但现有基于幅度的方法易误剪推理相关神经元:因过度依赖离散交叉熵目标,陷入‘幅度陷阱’,在高稀疏度下(如40%)出现推理能力崩溃。为克服这一拓扑相变,我们提出相对动能实用度(RKU),一种基于交替梯度流(AGF)的连续动能积分理论框架。通过引入费舍尔迹归一化,RKU作为轻量级曲率感知归一化,有效分离出‘动能尖峰’——即高曲率逻辑路由的关键结构通路。在Qwen-2.5-7B和LLaMA-3-8B上的大量实验表明,该方法在约40%稀疏度下性能显著提升,40%稀疏度下于GSM8K达到13.34%准确率,超越最强基线,并在分布外评估中更优地保留了推理相关表示。

原文摘要 · Abstract (English)

Chain-of-Thought (CoT) prompting symbolized a huge improvement of reasoning capabilities of Large Language Models (LLMs). However, scaling up test-time computation yields extensive CoT sequences, introducing severe inference latency and key-value (KV) cache memory bottlenecks. While structural pruning offers a fundamental, hardware-aware solution to alleviate static parameter burdens, existing magnitude-based methods may cut off the neurons of CoT: by over-indexing on discrete cross-entropy objectives, these heuristics fall into a \textit{magnitude trap}: they prioritize high-frequency, low-information syntactic tokens and trigger a disappointing reasoning collapse at high sparsities (e.g., 40\%). To overcome this topological phase transition, we propose \textsc{Relative Kinetic Utility} (RKU), a novel theoretical framework that elevates discrete pruning to a continuous kinetic integral over the depth manifold of the model based on Alternating Gradient Flow(AGF). By modifying it with Fisher trace normalization, RKU acts as a lightweight curvature-aware normalization to isolate \textit{kinetic spikes} -- the fundamental structural pathways responsible for high-curvature logical routing. Extensive experiments on Qwen-2.5-7B and LLaMA-3-8B improves performance in the high-sparsity regime around 40\%. RKU attains 13.34\% accuracy on GSM8K at 40\% sparsity, outperforming the strongest baseline, and appears to better preserve reasoning-relevant representations under out-of-distribution evaluation.

大模型剪枝推理优化链式思维结构化剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。