让大模型学会自我推理，通过可学习的提示词自动构建思考链条。
Reasoning with Reinforced Functional Token Tuning
- 引入可学习的功能性标记（如<分析><验证>），让模型自主生成推理过程。
- 在MATH数据集上，Qwen-2.5-7B-Instruct准确率从70.6%提升至79.8%。
- 支持无提示探索，适合需要强推理能力的数学与逻辑任务研究者。
本文提出强化功能标记微调（RFTT），一种新型强化微调框架，使大语言模型具备自对弈式学习推理的能力。不同于以往基于提示的推理方法，RFTT将一系列可学习的功能标记（如<分析><验证><优化>）直接嵌入模型词汇表，实现多样化类人推理行为的链式思考构建。具体包含两个阶段：(1) 监督微调通过提示驱动的树搜索生成带功能标记的自生成训练数据，使模型初步掌握这些标记的推理用法；(2) 在线强化学习允许模型通过功能标记采样自主探索不同推理路径，无需依赖外部提示，从而实现功能性推理的持续自我优化。大量实验表明，RFTT在数学基准测试中表现优异，显著提升Qwen-2.5-7B-Instruct（从70.6%到79.8%）和LLaMA-3.1-8B-Instruct（从32.2%到60.2%）在MATH数据集上的性能。此外，推理时更多搜索轮次能进一步提升性能。代码已开源。
原文摘要 · Abstract (English)
In this work, we propose Reinforced Functional Token Tuning (RFTT), a novel reinforced fine-tuning framework that empowers Large Language Models (LLMs) with self-play learn-to-reason capabilities. Unlike prior prompt-driven reasoning efforts, RFTT embeds a rich set of learnable functional tokens (e.g., <analyze>, <verify>, <refine>) directly into the model vocabulary, enabling chain-of-thought construction with diverse human-like reasoning behaviors. Specifically, RFTT comprises two phases: (1) supervised fine-tuning performs prompt-driven tree search to obtain self-generated training data annotated with functional tokens, which warms up the model to learn these tokens for reasoning; and (2) online reinforcement learning further allows the model to explore different reasoning pathways through functional token sampling without relying on prompts, thereby facilitating effective self-improvement for functional reasoning. Extensive experiments demonstrate the superiority of the proposed RFTT on mathematical benchmarks, significantly boosting Qwen-2.5-7B-Instruct (70.6% to 79.8%) and LLaMA-3.1-8B-Instruct (32.2% to 60.2%) on the MATH dataset. Moreover, the performance of RFTT consistently improves with more search rollouts at inference time. Our code is available at https://github.com/sastpg/RFTT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。