只微调关键推理步骤，让大模型更准更会变通。
Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
- 通过反事实扰动找出决定推理对错的关键词元，只对它们微调。
- 仅更新少于12%的词元，却在11个数学推理任务上全面超越标准微调。
- 适合追求高效、高多样性与强泛化能力的模型优化场景。
大语言模型主要依赖监督微调（SFT）来适应数学推理等特定任务，但标准SFT对所有词元一视同仁地惩罚，忽略了只有少数关键词元决定推理正确性。这种均匀监督常导致输出多样性下降和泛化能力受限。我们提出关键词元微调（CFT），一种简单有效的方法：通过反事实扰动识别出功能上不可或缺的词元，并仅对其更新梯度信号。通过聚焦于这些决定性推理步骤，同时保留非关键词元的多样性，CFT能提升生成质量与多样性。在三个模型家族（Qwen、OLMo、LLaMA）的五种模型上，对十一项数学推理基准的广泛实验表明，尽管仅微调不足12%的词元，CFT始终优于标准SFT。此外，CFT可通过改进采样多样性实现测试时扩展，并为强化学习提供更强初始化，在后续训练阶段维持性能优势，同时保持更高熵以促进更好探索。结果表明，CFT是一种高效且稳健的大模型微调通用框架。
原文摘要 · Abstract (English)
Large language models (LLMs) primarily rely on supervised fine-tuning (SFT) as a key method to adapt pre-trained models to domain-specific tasks such as mathematical reasoning. However, standard SFT uniformly penalizes all tokens, neglecting that only a small subset of critical tokens determines reasoning correctness. This uniform supervision often causes reduced output diversity and limited generalization. We propose Critical Token Fine-tuning (CFT), a simple yet effective approach that updates only tokens identified as functionally indispensable via counterfactual perturbations. By focusing gradient signals on these decisive reasoning steps while preserving the diversity of non-critical tokens, CFT can enhance both generation and diversity. Extensive experiments on five models across three families (Qwen, OLMo, LLaMA) and eleven mathematical reasoning benchmarks show that CFT, despite fine-tuning on less than 12% of tokens, consistently outperforms standard SFT. Moreover, CFT enables test-time scaling through improved sampling diversity and provides a stronger initialization for reinforcement learning, sustaining performance gains in later training stages while maintaining higher entropy for better exploration. These results highlight CFT as a practical and general framework for efficient and robust LLM fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。