通过激活向量调控,让大模型推理更简洁高效。
Activation Steering for Chain-of-Thought Compression
- 用激活空间中的转向向量控制推理风格,实现无训练压缩。
- 仅用100对样本,使推理长度减少67.43%,准确率不变。
- 无需重训练,推理速度提升2.73倍,适合部署优化。
大型语言模型在包含中间推理步骤(链式思维,CoT)时表现优异,但这些推理过程常过于冗长,尤其对简单问题,导致上下文浪费、延迟增加和能耗上升。我们发现,冗长的英文主导型CoT与简洁的数学主导型CoT在模型残差流激活空间中占据不同区域。通过提取并注入一个“转向向量”,可稳定地引导生成向更简洁模式转换,从而在不重新训练的情况下有效压缩CoT。我们提出激活引导压缩(ASC),一种推理阶段直接修改隐藏表示的技术。此外,我们从闭式表达推导出基于KL散度约束的理论分析,以调节转向强度。仅使用100对冗长与简洁示例,ASC在MATH500和GSM8K数据集上实现最高67.43%的CoT长度缩减,且在7B、8B、32B参数模型上保持准确率。作为免训练方法,ASC引入极小运行时开销,在8B模型上使端到端推理耗时平均提速2.73倍。该方法适用于低延迟或低成本场景下的推理模型部署。代码已开源:https://github.com/ArminAzizi98/ASC。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at complex reasoning when they include intermediate steps, known as "chains of thought" (CoTs). However, these rationales are often overly verbose, even for simple problems, leading to wasted context, increased latency, and higher energy consumption. We observe that verbose, English-heavy CoTs and concise, math-centric CoTs occupy distinct regions in the model's residual-stream activation space. By extracting and injecting a "steering vector" to transition between these modes, we can reliably shift generation toward more concise reasoning, effectively compressing CoTs without retraining. We formalize this approach as Activation-Steered Compression (ASC), an inference-time technique that shortens reasoning traces by directly modifying hidden representations. In addition, we provide a theoretical analysis of the impact of ASC on the output distribution, derived from a closed-form KL-divergence-bounded constraint to regulate steering strength. Using only 100 paired verbose and concise examples, ASC achieves up to 67.43% reduction in CoT length on MATH500 and GSM8K datasets, while maintaining accuracy across 7B, 8B, and 32B parameter models. As a training-free method, ASC introduces negligible runtime overhead and, on MATH500, delivers an average 2.73x speedup in end-to-end reasoning wall-clock time on an 8B model. This makes ASC a practical and efficient tool for streamlining the deployment of reasoning-capable LLMs in latency- or cost-sensitive settings. The code is available at: https://github.com/ArminAzizi98/ASC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。