比较六种离线推理训练方法的权重更新方向，发现它们本质差异显著。
Weight-Space Geometry of Offline Reasoning Training
- 通过余弦相似度等几何分析，量化不同方法的权重更新方向
- DPO效果最好（GSM8K 93.5%），但更新方向几乎垂直于其他方法
- 即使数据相同，方法差异导致权重更新路径大不相同
离线强化学习损失函数（RFT、RIFT、DFT、Offline GRPO、DPO）被广泛用于将大模型的推理能力蒸馏到小模型，通常仅以下游准确率进行比较。本文在单一基础模型（Qwen3-4B）的数学轨迹上，使用仅注意力的LoRA，训练六种方法（SFT、RFT、DFT、RIFT、Offline GRPO、DPO），通过余弦相似度、主角子空间分析、线性模式连通性和CKA分析权重变化。结果表明：(i) SFT、RFT和RIFT的权重增量几乎共线（余弦相似度≥0.97，前1主角中位数约7度，覆盖144个模块），且在GSM8K上准确率相近（87–88%，n=1319；配对McNemar p ≥ 0.15）；(ii) DFT虽使用相同数据，方向仍显著偏离其他奖励加权方法；(iii) Offline GRPO在全局约67%（晚期层达86%）的更新分量与SFT正交，但仍处于SFT损失盆地内；(iv) DPO位于近乎正交的子空间，存在模式连通性屏障，晚期层的CKA降至约0.46。DPO在本协议下表现最优，GSM8K达93.5%（相比其他方法，McNemar p < 10^-9），AIME26达30.0%（对比3.3–10.0%）；其训练采用10倍更小的学习率（标准惯例），因此更新范数与准确率差异反映损失函数与优化器共同作用，学习率匹配下的对比留待未来工作。
原文摘要 · Abstract (English)
Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone. We ask whether they are mechanistically distinct or converge to a similar weight update. Training six methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical math rollouts from a single base model (Qwen3-4B) with attention-only LoRA, we analyze the resulting deltas via cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA. We observe: (i) SFT, RFT, and RIFT have nearly colinear weight deltas (cosine >= 0.97, top-1 principal angle ~7 deg median over 144 modules) and comparable GSM8K accuracy (87-88%, n=1319; pairwise McNemar p >= 0.15); (ii) DFT diverges further in direction than any reward-weighted method despite using the same data; (iii) Offline GRPO adds a substantial component orthogonal to the SFT direction (~67% globally, up to ~86% in late layers) while staying in the SFT loss basin; (iv) DPO sits in a near-orthogonal subspace, shows a mode-connectivity barrier, and collapses late-layer CKA to ~0.46. DPO also reaches the highest accuracy in our protocol on both GSM8K (93.5%, McNemar p < 10^-9 vs. each other method) and AIME26 (30.0% vs. 3.3-10.0%); its training uses a 10x smaller learning rate than the others (the standard convention), so the update-norm and accuracy gaps reflect loss-function and optimizer choices jointly, and a learning-rate-matched DPO comparison is left for future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。