通过去除推理过程中的思考内容，提升强化学习推理模型的效率与性能。
Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
- 引入无思考策略初始化，训练时直接丢弃思考内容以减少计算开销。
- 4B模型在不到4000小时的训练下，AIME24准确率达89.0%，LiveCodeBench达65.5%。
- 无需特殊奖励或复杂设计，显著加速收敛并提升推理效率，适合高效推理系统构建。
基于可验证奖励的强化学习（RLVR）虽能有效解决复杂任务，但训练需极长上下文，导致巨大计算成本。多阶段训练可部分缓解，但起始上下文过短常引发不可逆性能下降，难以显著降低总训练算力。本文提出思维自由策略初始化（TFPI），一种简单而有效的RLVR改进方法，连接长链式思维（CoT）蒸馏与标准RLVR。TFPI采用简单的*ThinkFree*操作，通过直接追加*</think>*显式丢弃思考内容，从而在推理中降低令牌消耗。使用*ThinkFree*适配输入进行训练，可提升性能并减少令牌使用量，即使在原始慢思考模式下亦然。大量实验表明，TFPI加速了RL收敛，达到更高性能上限，并生成更高效的推理模型，无需专用奖励或复杂训练设计。仅用TFPI，我们训练出的4B模型在AIME24上达到89.0%准确率，在LiveCodeBench上达65.5%，训练耗时低于4000小时。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce **T**hinking-**F**ree **P**olicy **I**nitialization (**TFPI**), a simple yet effective adaptation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. TFPI employs a simple *ThinkFree* operation, explicitly discarding the thinking content via a direct *</think>* append, to reduce token usage during inference. Training with *ThinkFree*-adapted inputs improves performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that TFPI accelerates RL convergence, achieves a higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train a 4B model to reach 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。