用极小的可训练参数实现大模型推理能力提升
Steering LLM Reasoning Through Bias-Only Adaptation
- 每层仅训练一个低维控制向量，冻结原模型权重
- 80亿参数模型仅增加0.0016%参数，性能媲美全量微调
- 适合追求高效推理优化的研究者与工程团队
我们发现，通过强化学习在每一层仅训练一个d维控制向量，同时冻结所有基础权重，即可在数学推理任务上达到全量强化学习微调模型的准确率。在80亿参数模型上，该方法仅增加约0.0016%的额外参数，并在多种基础模型和数学推理基准上复现了相近性能。这一结果收紧了高阶思维链推理所需的参数预算上限，表明数百万个适配器权重实属冗余。极小的可训练规模降低了优化器内存占用与跨GPU通信开销，显著降低微调成本。此外，逻辑值透镜分析显示，学习到的向量增强了连贯的词元方向，为模型内部计算提供了更清晰的解释。
原文摘要 · Abstract (English)
We show that training a single $d$-dimensional steering vector per layer with reinforcement learning, while freezing all base weights, matches the accuracy of fully RL-tuned reasoning models on mathematical-reasoning tasks. On an 8 billion-parameter model this adds only $\approx 0.0016\%$ additional parameters and reproduces performance across a range of base models and mathematical-reasoning benchmarks. These results tighten the upper bound on the parameter budget required for high-level chain-of-thought reasoning, indicating that millions of adapter weights are unnecessary. The minimal trainable footprint reduces optimizer memory and inter-GPU communication, lowering the overall cost of fine-tuning. Moreover, a logit-lens analysis shows that the learned vectors amplify coherent token directions, providing clearer insight into the model's internal computations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。