针对推理链训练中令牌权重不均的问题,提出自适应重加权方法提升模型泛化能力。
VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
- 将推理链监督建模为约束优化问题,动态分配不同令牌的训练权重
- 在数学与编程任务上显著提升性能,尤其对小模型提升明显
- 可作为强化学习的更好初始化,适合提升大模型推理能力
在长推理链(CoT)轨迹上进行监督微调(SFT)已成为提升大语言模型(LLM)推理能力的关键技术。然而,标准交叉熵损失对所有令牌一视同仁,忽略了其在推理过程中贡献的异质性,导致监督分配失衡,尤其在复杂长推理任务中影响模型泛化。为此,我们提出基于优化理论的方差可控重加权框架(VCORE),将CoT监督重新表述为一个约束优化问题。通过优化视角,实现令牌间训练权重的自适应分配,使训练目标更贴近鲁棒推理泛化的本质。实验证明,VCORE在整体平均性能上表现最优,尤其在低容量模型上增益显著。在数学与编程基准测试中,使用Qwen3系列(4B、8B、32B)和LLaMA-3.1-8B-Instruct模型均取得显著提升。此外,我们证明了VCORE能为后续强化学习提供更优初始化,为推进大模型推理能力奠定坚实基础。代码将发布于 https://github.com/coder-gx/VCORE。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) on long chain-of-thought (CoT) trajectories has emerged as a crucial technique for enhancing the reasoning abilities of large language models (LLMs). However, the standard cross-entropy loss treats all tokens equally, ignoring their heterogeneous contributions across a reasoning trajectory. This uniform treatment leads to misallocated supervision and weak generalization, especially in complex, long-form reasoning tasks. To address this, we introduce \textbf{V}ariance-\textbf{C}ontrolled \textbf{O}ptimization-based \textbf{RE}weighting (VCORE), a principled framework that reformulates CoT supervision as a constrained optimization problem. By adopting an optimization-theoretic perspective, VCORE enables a principled and adaptive allocation of supervision across tokens, thereby aligning the training objective more closely with the goal of robust reasoning generalization. Empirical evaluations demonstrate that VCORE achieves the strongest overall average performance, with especially clear gains on lower-capacity models. Across both in-domain and out-of-domain settings, VCORE achieves substantial performance gains on mathematical and coding benchmarks, using models from the Qwen3 series (4B, 8B, 32B) and LLaMA-3.1-8B-Instruct. Moreover, we show that VCORE serves as a more effective initialization for subsequent reinforcement learning, establishing a stronger foundation for advancing the reasoning capabilities of LLMs. The Code will be released at https://github.com/coder-gx/VCORE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。