用熵调控损失函数,解释为何先微调再强化学习才有效。
How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

- 通过Tsallis损失连续体统一建模微调与强化学习的顺序
- 高q值快速逃离冷启动,低q值抗噪声但易过拟合
- 无需推理标注即可训练,适合需要稳定性的问答任务
SFT-then-RLVR是推理模型后训练的常用范式,但其顺序选择及纯强化学习在冷启动时失效的原因缺乏统一理论。本文基于包含Tsallis q-对数的统一损失族$J_Q$提供解释:该族单参数插值于RLVR(q=0,exploitation极点)与隐轨迹对数似然(q=1,density-estimation极点)之间,标准流程对应于q从1到0的分步调度。所有成员共享相同梯度方向,仅以$P_θ^{-q}$按实例独立放大。在梯度流分析下,exploitation极点需Ω(1/p₀)时间逃离冷启动,但抗标签噪声;density-estimation极点仅需Θ(log(1/p₀))即逃逸,但会记忆噪声。此分离解释了SFT(q=1)先移出冷启动区、再用更鲁棒的RLVR(q=0)的机制。进一步提出两种无标注推理的蒙特卡洛估计器:梯度增强的RL(GARL)与后验衰减微调(PAFT),具有相同偏差$O(q/(M P_θ^q))$但差异化的方差与稳定性。在FinQA、HotPotQA和MuSiQue上,GARL在高q值下显著缓解冷启动停滞,优于GRPO;在热启动下,低q值的GARL主导FinQA;在HotPotQA和MuSiQue上,GARL不稳定,而PAFT在q=0.75时保持稳定,达到47.9 m@16(比GRPO高13.9)。
原文摘要 · Abstract (English)
SFT-then-RLVR is widely used for post-training reasoning models, but why this specific ordering, and why RLVR-only stalls at cold start, have lacked a unifying theoretical account. We provide that account under a unified loss family $J_Q$ using the Tsallis $q$-logarithm. $J_Q$ is a single-parameter family that interpolates between RLVR (at $q{=}0$, the \textit{exploitation pole}) and the log-marginal-likelihood over latent trajectories (at $q{=}1$, the \textit{density-estimation pole}), under which the standard pipeline corresponds to a stepwise $q{=}1 \to 0$ schedule. All members share the same per-example gradient direction, differing only by a per-instance amplification $P_θ^{-q}$ that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires $Ω(\frac{1}{p_0})$ time to escape cold start but is robust to label noise, while the density-estimation pole escapes in $Θ\big(\log(\frac{1}{p_0})\big)$ but memorizes label noise. This separation explains how SFT ($q{=}1$) first moves the model out of the cold-start regime, followed by the more robust RLVR ($q{=}0$), under the SFT-then-RLVR paradigm. We further derive two Monte Carlo estimators that directly optimize fixed-$q$ on the $J_Q$ continuum, without annotated rationales: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias $O\big(\frac{q}{M P_θ^q}\big)$ but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high $q$ substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low $q$ dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes and PAFT at $q{=}0.75$ remains stable, reaching $47.9$ \texttt{m@16} on HotPotQA ($+13.9$ over GRPO).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。