用向量转移大模型的推理能力,无需重训即可提升性能。
Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
- 从不同训练方式的模型中提取推理向量,通过加法注入新模型。
- 在多个推理任务上表现显著提升,最高增益达12.3%。
- 适合想低成本提升模型推理能力的研究者和开发者。
大型语言模型通常需要昂贵的优化(如强化学习)才能掌握复杂推理任务。本文表明,一旦学会推理能力,便可作为紧凑的任务向量在模型间迁移。我们使用两个相同初始化的Qwen2.5模型:一个经监督微调(SFT),另一个经组相对策略优化(GRPO)训练于同一数据集。从中提取推理向量:$v_{\text{reason}} = θ_{\text{GRPO}} - θ_{\text{SFT}}$。该向量假设捕捉了强化学习引入的推理能力,同时剔除了SFT共享的知识。将此向量通过简单算术添加至兼容的指令微调模型后,可在多个推理基准上持续提升性能:GSM8K (+4.9%)、HumanEval (+4.3%)、SciQ (+1.7%),BigBenchHard上1.5B模型提升12.3%。性能提升在对抗性条件下依然有效。反之,减去该向量导致显著退化(GSM8K下降-11.8%),证明其对推理能力的关键作用。本工作展示了如何从已有开源模型中提取推理能力,并通过简单的张量运算复用,为高效增强模型提供可行路径。
原文摘要 · Abstract (English)
Large language models often require costly optimization, such as reinforcement learning, to master complex reasoning tasks. This work demonstrates that reasoning ability, once learned, can be extracted and transferred between models as a compact task vector. We source two publicly available, identically initialized Qwen2.5 models, one fine-tuned with supervised fine-tuning (SFT) and the other with group relative policy optimization (GRPO) on the same dataset. From these, we extract a reasoning vector: $v_{\text{reason}} = θ_{\text{GRPO}} - θ_{\text{SFT}}$. We hypothesize that this vector captures the reasoning capability instilled by reinforcement learning while factoring out shared knowledge from the SFT process. When added to compatible instruction-tuned models through simple arithmetic, this vector consistently improves performance across diverse reasoning benchmarks: GSM8K (+4.9%), HumanEval (+4.3%), SciQ (+1.7%), and BigBenchHard (+12.3% for the 1.5B model). The performance improvements persist under adversarial conditions. Conversely, subtracting the vector causes significant performance degradation (-11.8% on GSM8K), demonstrating the vector's strong contribution to the model's reasoning abilities. This work shows how reasoning capabilities, typically developed through expensive training, can be extracted from existing open-source models and reused through simple tensor arithmetic, offering a practical way to enhance models by recycling prior computational investments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。