用紧凑向量迁移大模型的推理能力，成本低效果好。
CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
- 提出CoT向量，用小参数编码多步推理知识。
- 在多个任务上表现接近微调，且参数更少。
- 可作为探针揭示模型推理机制的内在规律。
链式思维（CoT）提示已成为提升大语言模型推理能力的有效方法。然而，现有方法如上下文学习和微调仍成本高昂且效率低下。为以更低代价提升CoT推理能力，受任务向量范式的启发，我们提出CoT向量——一种编码通用、多步推理知识的紧凑表示。通过实验发现，提取的CoT向量存在显著的层间不稳定性，表现为典型的U型性能曲线，反映了模型中系统性的三阶段推理过程。为此，我们提出可学习的CoT向量，在教师-学生框架下优化，提供更稳定可靠的引导。在多种基准和模型上的广泛评估表明，CoT向量不仅优于现有基线，还达到与参数高效微调相当的性能，同时所需可训练参数更少。此外，将CoT向量作为探针，我们揭示其有效性受潜在空间结构、信息密度、获取机制及预训练差异的影响，为多步推理在大模型中的功能组织提供了新见解。源代码将公开。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing implementations, such as in-context learning and fine-tuning, remain costly and inefficient. To improve CoT reasoning at a lower cost, and inspired by the task vector paradigm, we introduce CoT Vectors, compact representations that encode task-general, multi-step reasoning knowledge. Through experiments with Extracted CoT Vectors, we observe pronounced layer-wise instability, manifesting as a U-shaped performance curve that reflects a systematic three-stage reasoning process in LLMs. To address this limitation, we propose Learnable CoT Vectors, optimized under a teacher-student framework to provide more stable and robust guidance. Extensive evaluations across diverse benchmarks and models demonstrate that CoT Vectors not only outperform existing baselines but also achieve performance comparable to parameter-efficient fine-tuning methods, while requiring fewer trainable parameters. Moreover, by treating CoT Vectors as a probe, we uncover how their effectiveness varies due to latent space structure, information density, acquisition mechanisms, and pre-training differences, offering new insights into the functional organization of multi-step reasoning in LLMs. The source code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。