arXiv:2505.17697cs.CLcs.LG2025-05NeurIPS被引 9

不训练也能让大模型长思考,靠调控关键激活值。

Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models

  • 通过放大最后几层的高影响激活值并插入等待标记,实现无训练长链条推理。
  • 使模型自我反思率和准确率显著提升,且激活变化有可预测规律。
  • 适合想低成本提升模型推理能力的研究者或开发者使用。

尽管大型语言模型展现出出色的推理能力,但激发其长链式思维(CoT)通常需要昂贵的强化学习或高质量蒸馏数据的监督微调。我们研究了该能力的内在机制,发现最后几层中少量高影响激活值主导了长推理特性,如输出长度和自我反思。仅通过放大这些激活值并插入“wait”标记,即可在不进行任何训练的情况下触发长链式思维,显著提升自我反思率与准确性。此外,我们发现激活动态遵循可预测轨迹:特殊标记后出现急剧上升,随后呈指数衰减。基于此,我们提出一种通用的免训练激活控制方法,利用少量对比样本识别关键激活值,并在推理时用简单解析函数调节其数值以激发长链式思维。大量实验验证了该方法在高效激发长链式推理及提升性能上的有效性。同时,我们还提出一种参数高效的微调方法,仅训练最后一层激活放大模块和少数LoRA层,在推理基准上表现优于全量LoRA微调,且参数更少。代码与数据已公开。

原文摘要 · Abstract (English)

Despite the remarkable reasoning performance, eliciting the long chain-of-thought (CoT) ability in large language models (LLMs) typically requires costly reinforcement learning or supervised fine-tuning on high-quality distilled data. We investigate the internal mechanisms behind this capability and show that a small set of high-impact activations in the last few layers largely governs long-form reasoning attributes, such as output length and self-reflection. By simply amplifying these activations and inserting "wait" tokens, we can invoke the long CoT ability without any training, resulting in significantly increased self-reflection rates and accuracy. Moreover, we find that the activation dynamics follow predictable trajectories, with a sharp rise after special tokens and a subsequent exponential decay. Building on these insights, we introduce a general training-free activation control technique. It leverages a few contrastive examples to identify key activations, and employs simple analytic functions to modulate their values at inference time to elicit long CoTs. Extensive experiments confirm the effectiveness of our method in efficiently eliciting long CoT reasoning in LLMs and improving their performance. Additionally, we propose a parameter-efficient fine-tuning method that trains only a last-layer activation amplification module and a few LoRA layers, outperforming full LoRA fine-tuning on reasoning benchmarks with significantly fewer parameters. Our code and data are publicly released.

长链推理激活调控免训练LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。