arXiv:2506.11274cs.CLcs.LG2025-06被引 3

学会自动延续思考的标记,让模型推理更准。

Learning a Continue-Thinking Token for Enhanced Test-Time Scaling

  • 用强化学习训练一个可学习的继续思考标记
  • 在GSM8K上比固定标记提升4.2%准确率
  • 适合需要高精度推理的应用场景

测试时扩展已成为通过增加推理阶段计算量来提升语言模型性能的有效方法。近期研究发现,用固定标记(如替换"</think>"为"Wait")可延长推理步骤并提高准确率。本文探索是否可学习一个专用的继续思考标记以触发更长推理过程。我们在深度蒸馏版DeepSeek-R1基础上,加入一个可学习的'<|continue-thinking|>'标记,仅通过强化学习训练其嵌入向量,保持模型权重不变。实验表明,该学习标记在标准数学基准测试中优于基线模型及使用固定标记的测试时扩展方法。尤其在固定标记能提升基线准确率的情况下,我们的方法提升更为显著:例如在GSM8K上,固定标记仅带来1.3%绝对提升,而本方法实现4.2%的提升。

原文摘要 · Abstract (English)

Test-time scaling has emerged as an effective approach for improving language model performance by utilizing additional compute at inference time. Recent studies have shown that overriding end-of-thinking tokens (e.g., replacing "</think>" with "Wait") can extend reasoning steps and improve accuracy. In this work, we explore whether a dedicated continue-thinking token can be learned to trigger extended reasoning. We augment a distilled version of DeepSeek-R1 with a single learned "<|continue-thinking|>" token, training only its embedding via reinforcement learning while keeping the model weights frozen. Our experiments show that this learned token achieves improved accuracy on standard math benchmarks compared to both the baseline model and a test-time scaling approach that uses a fixed token (e.g., "Wait") for budget forcing. In particular, we observe that in cases where the fixed-token approach enhances the base model's accuracy, our method achieves a markedly greater improvement. For example, on the GSM8K benchmark, the fixed-token approach yields a 1.3% absolute improvement in accuracy, whereas our learned-token method achieves a 4.2% improvement over the base model that does not use budget forcing.

推理增强测试时扩展强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。