通过推理时插入填充标记，扩展大模型计算空间，提升小模型性能。
Expanding Computation Spaces of LLMs at Inference Time
- 推理时插入特定填充标记，扩大模型计算空间。
- 小模型（如1.7B）性能提升最高达12.372个百分点。
- 适用于需增强推理能力的轻量级模型部署场景。
链式思维（CoT）推理使语言模型能利用额外的任务相关文本进行问题求解，不仅受益于详细的推理步骤，还得益于更长输入带来的计算空间扩展。以往工作通过训练填充或特殊标记作为额外计算空间。本研究探究语言模型是否能在推理阶段仅通过人工插入填充标记来利用这些空间。我们首先识别有效的标记类型、数量和插入位置，接着分析模型在训练哪个阶段开始利用扩展计算空间，并通过注意力图谱分析这些空间内的动态变化。在涵盖1.7B至32B参数规模的开放域问答与数学任务上的实验表明，合适的标记类型和数量因模型而异，但将填充标记直接置于最终'Answer:'前最为有效。小模型收益最大，SmolLM2-1.7B-Instruct提升高达12.372个百分点，表明这些空间提供了额外计算容量而非冗余输入。注意力图谱显示，扩展空间常延续原有注意力机制，有时聚焦于问题或答案选项，说明其对问题求解具有实际意义。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) rationale enables language models to use additional task-related text for problem-solving, benefiting not only from detailed reasoning steps but also from the expanded computational space of longer inputs. Prior work has trained filler or special tokens to serve as additional computation spaces. In this study, we investigate whether language models can leverage artificially inserted sequences of filler tokens solely at inference. We first identify effective token types, numbers, and insertion locations, then examine at what stage of training models begin to exploit the expanded computation space, and finally analyze dynamics within these spaces via attention maps. Experiments on models ranging from 1.7B to 32B across open-domain QA and math tasks show that appropriate token types and counts vary, but placing filler tokens directly before the final 'Answer:' token is most effective. Smaller models benefit most, up to 12.372 percentage points in SmolLM2-1.7B-Instruct, indicating that these spaces act as additional computational capacity rather than redundant input. Attention maps reveal that expanded spaces often continue the original attention mechanism and sometimes focus on questions or answer options, suggesting meaningful computation for problem-solving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。