小模型在极低预算下也能实现强推理，关键在于适配器秩与初始化的平衡。
Plasticity vs. Rigidity: The Impact of Low-Rank Adapters on Reasoning on a Micro-Budget
- 用低秩适配器和强化学习，在单块A40上24小时内训练15亿参数以下模型。
- 最优结果在AIME 24上达40.0%准确率，比基线提升11.1%，链式思维能力显著增强。
- 高秩适配器让模型更灵活，但数学对齐过强的模型反而因噪声更新而崩溃。
当前数学推理研究多依赖大规模模型，但小模型（≤1.5B）在极端约束下是否仍具备强推理能力尚不明确。本文在单张A40 GPU（48GB）上，仅用24小时，通过基于可验证奖励的强化学习（RLVR）与低秩适配（LoRA）训练小语言模型。结果表明，微预算训练的成功取决于适配器容量与模型初始化的相互作用：当适配器秩r=8时，无法捕捉复杂推理优化动态；而当r=256时，标准指令微调模型展现出显著可塑性。最佳模型在AIME 24上达到40.0% Pass@1，较基线绝对提升11.1%，且Pass@16达70.0%，体现强大探索能力。然而，这种可塑性并非普适：指令微调模型能利用预算扩展思维链以最大化奖励，而高度数学对齐的模型则出现性能崩溃，表明低预算强化学习的噪声更新可能对已接近任务最优的模型造成破坏性干扰。
原文摘要 · Abstract (English)
Recent advances in mathematical reasoning typically rely on massive scale, yet the question remains: can strong reasoning capabilities be induced in small language models ($\leq1.5\text{B}$) under extreme constraints? We investigate this by training models on a single A40 GPU (48GB) for under 24 hours using Reinforcement Learning with Verifiable Rewards (RLVR) and Low-Rank Adaptation (LoRA). We find that the success of this ``micro-budget" regime depends critically on the interplay between adapter capacity and model initialization. While low-rank adapters ($r=8$) consistently fail to capture the complex optimization dynamics of reasoning, high-rank adapters ($r=256$) unlock significant plasticity in standard instruction-tuned models. Our best result achieved an impressive 40.0\% Pass@1 on AIME 24 (an 11.1\% absolute improvement over baseline) and pushed Pass@16 to 70.0\%, demonstrating robust exploration capabilities. However, this plasticity is not universal: while instruction-tuned models utilized the budget to elongate their chain-of-thought and maximize reward, heavily math-aligned models suffered performance collapse, suggesting that noisy, low-budget RL updates can act as destructive interference for models already residing near a task-specific optimum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。