动态插入[PAUSE] token提升大模型推理能力
Learning to Insert [PAUSE] Tokens for Better Reasoning
- 根据模型置信度低的位置动态插入[PAUSE] token
- 在GSM8K上最高提升4.7个百分点准确率
- 适合需要增强逻辑推理的LLM研究者
为提升推理能力，已有工作尝试在训练中引入专用标记。受此启发，本文提出动态插入标记训练（DIT）方法：基于词元对数似然判断序列中模型置信度最低的位置，针对性插入[PAUSE]标记，从而增强后续词元预测能力。在多个数据集和模型（2.7B至8B参数规模）上的实验表明，DIT consistently优于传统微调及先前标记插入方法，在GSM8K上准确率提升最高达4.7%p，AQUA-RAT提升3.23%p，MBPP pass@1最高提升3.4%p。该方法基于模型自身表现而非人工设定，拓展了推理增强的研究边界。
原文摘要 · Abstract (English)
To enhance reasoning capabilities, previous works have explored incorporating special-purpose tokens into the training process. These strategies strengthen the learning mechanism of transformer-based large language models (LLMs). Building on prior research, in which inserting dummy tokens consecutively just before reasoning steps can enhance effectiveness, we introduce a novel approach termed Dynamic Inserting Tokens Training (DIT). Our method identifies positions within sequences where model confidence is lowest according to token log-likelihood. Strategically inserting [PAUSE] tokens on these positions bolsters the model's predictive capabilities for subsequent tokens. Experimental results across diverse datasets and models, from the 2.7B model to the 8B model, demonstrate that DIT consistently outperforms traditional fine-tuning and previous token insertion methods. With this simple yet effective method, we achieve accuracy gains of up to 4.7%p on GSM8K, 3.23%p on AQUA-RAT, and pass@1 improvements of up to 3.4%p on MBPP datasets. Our work shows a model-based, dynamic approach rather than a heuristic one, thereby broadening the scope of research in reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。