让大模型同时解多个题,自动省token还更准。
Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
- 训练模型一次处理多个问题,靠准确率奖励自动控长
- 问题数越多,每题用的词越少,准确率下降也更平缓
- 无需复杂训练,就能在数学题上省6成以上token
采用思维链推理的大语言模型表现优异,但推理过程消耗大量词元,推高了推理成本。现有效率方法如显式长度惩罚、难度估计或分阶段课程训练,要么降低推理质量,要么需要复杂的训练流程。我们提出批处理上下文强化(Batched Contextual Reinforcement, BCR),一种极简的单阶段训练范式,通过一个简单结构改动——在共享上下文窗口内同时训练模型解决N个问题,仅以每题准确率为奖励——实现高效推理。该设定产生隐式词元预算,带来若干关键发现:(1) 我们发现一种新型任务扩展规律:推理时并发问题数N增加,单题词元使用量单调下降,而准确率退化远比基线温和,确立了N作为可控吞吐维度;(2) BCR打破了传统准确率-效率权衡,在标准单题推理下呈现“免费午餐”现象:在1.5B和4B模型族中,词元消耗减少15.8%至62.6%,且在五个主要数学基准上准确率保持或提升;(3) 定性分析显示模型自发消除冗余元认知循环,展现自调节效率;(4) 实证表明,隐式预算约束成功规避了显式长度惩罚固有的对抗梯度与灾难性优化坍缩问题,提供了一种高度稳定的长度控制方案。结果证明BCR实用性强,仅通过简单结构激励即可释放大模型中潜在的高密度推理能力。
原文摘要 · Abstract (English)
Large Language Models employing Chain-of-Thought reasoning achieve strong performance but suffer from excessive token consumption that inflates inference costs. Existing efficiency methods such as explicit length penalties, difficulty estimators, or multi-stage curricula either degrade reasoning quality or require complex training pipelines. We introduce Batched Contextual Reinforcement, a minimalist, single-stage training paradigm that unlocks efficient reasoning through a simple structural modification: training the model to solve N problems simultaneously within a shared context window, rewarded purely by per-instance accuracy. This formulation creates an implicit token budget that yields several key findings: (1) We identify a novel task-scaling law: as the number of concurrent problems N increases during inference, per-problem token usage decreases monotonically while accuracy degrades far more gracefully than baselines, establishing N as a controllable throughput dimension. (2) BCR challenges the traditional accuracy-efficiency trade-off by demonstrating a "free lunch" phenomenon at standard single-problem inference. Across both 1.5B and 4B model families, BCR reduces token usage by 15.8% to 62.6% while consistently maintaining or improving accuracy across five major mathematical benchmarks. (3) Qualitative analyses reveal emergent self-regulated efficiency, where models autonomously eliminate redundant metacognitive loops without explicit length supervision. (4) Crucially, we empirically demonstrate that implicit budget constraints successfully circumvent the adversarial gradients and catastrophic optimization collapse inherent to explicit length penalties, offering a highly stable, constraint-based alternative for length control. These results prove BCR practical, showing simple structural incentives unlock latent high-density reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。