通过调控模型激活值,无需训练即可提升大模型的推理能力。
Improving Reasoning Performance in Large Language Models via Representation Engineering
- 从残差流提取激活值生成控制向量,推理时干预模型表征空间。
- 在归纳、演绎和数学推理任务上,对Mistral-7B-Instruct和Pythia系列有显著性能提升。
- 方法简单有效,适用于希望改进特定任务推理表现的研究者或开发者。
大语言模型(LLMs)在推理能力方面取得进展,但其推理机制是否本质不同仍存在争议。本文提出一种表征工程方法:在模型处理推理任务时,从残差流读取激活值,生成控制向量,在推理阶段干预模型,调节其表征空间以提升特定任务表现。我们公开了生成控制向量与分析模型表征的代码。该方法在归纳、演绎和数学推理任务上应用于Mistral-7B-Instruct及多个Pythia模型,显著提升性能。通过KL散度与熵等指标分析,验证了控制向量对模型最终输出分布的影响。结果表明,推理能力可通过残差流干预实现可控调节,且无需额外训练即可改善特定任务表现。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have resulted in increasingly anthropomorphic language concerning the ability of LLMs to reason. Whether reasoning in LLMs should be understood to be inherently different is, however, widely debated. We propose utilizing a representation engineering approach wherein model activations are read from the residual stream of an LLM when processing a reasoning task. The activations are used to derive a control vector that is applied to the model as an inference-time intervention, modulating the representational space of the model, to improve performance on the specified task. We publish the code for deriving control vectors and analyzing model representations. The method allows us to improve performance on reasoning benchmarks and assess how control vectors influence the final logit distribution of a model via metrics such as KL divergence and entropy. We apply control vectors to Mistral-7B-Instruct and a range of Pythia models on an inductive, a deductive and mathematical reasoning task. We show that an LLM can, to a certain degree, be controlled to improve its perceived reasoning ability by modulating activations. The intervention is dependent upon the ability to reliably extract the model's typical state when correctly solving a task. Our results suggest that reasoning performance can be modulated in the same manner as other information-processing tasks performed by LLMs and demonstrate that we are capable of improving performance on specific tasks via a simple intervention on the residual stream with no additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。