分离训练再合成,用任务向量实现低成本高效模型增强。
Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
- 先独立训练SFT与RLVR,推理时通过任务向量算术融合
- 在多个数学推理基准上性能媲美甚至超过联合训练方法
- 仅需约3%计算成本,适合部署强后训练模型
SFT和RLVR是大模型后训练的两种基础范式,分别拓展知识广度与提升推理深度。但直接整合二者面临灾难性遗忘和梯度冲突问题。我们通过任务向量分析发现:两者存在30倍的幅度差异、45度的符号干扰及模块级更新分布异质性。这表明二者修改的是模型中部分互补组件。为此提出解耦测试时合成(DoTS)框架:允许SFT与RLVR独立训练,仅在推理时通过任务向量算术融合,不更新参数。为减少干扰,采用保持范数的稀疏化,并使用小规模无标签查询的贝叶斯优化,在一致性与困惑度的帕累托前沿搜索最优组合系数。实验显示, ext{ours} 在多个数学推理基准上达到或超越基于训练的集成方法性能,计算成本仅为约3%。应用于更强后训练检查点时,优于当前最优模型,且无需调参即可泛化至域外数据集。代码已开源。
原文摘要 · Abstract (English)
SFT and RLVR represent two fundamental yet distinct paradigms for LLM post-training, each excelling in distinct dimensions. SFT expands knowledge breadth while RLVR enhances reasoning depth. Yet integrating these complementary strengths remains a formidable challenge. Sequential training can cause catastrophic forgetting, and joint optimization often suffers from severe gradient conflicts. We analyze SFT and RLVR through the lens of task vectors and reveal three structural properties behind these failures: a 30* magnitude disparity, 45* sign interference, and heterogeneous module-wise update distributions. These findings show SFT and RLVR are difficult to integrate directly, but they also suggest that the two paradigms modify partly complementary components of the model. Motivated by these observations, we propose Decoupled Test-time Synthesis (DoTS), a post-hoc framework allows SFT and RLVR checkpoints to be trained independently and synthesizes their capabilities only at inference time via task vector arithmetic, without updating model parameters. To reduce interference, DOTS applies selective sparsification with norm-preserving rescaling. It then uses Bayesian optimization on a small set of unlabeled queries to search for combination coefficients on the Pareto frontier of consistency and perplexity. Empirically, \ours matches or exceeds the performance of training-based SFT--RLVR integration methods across multiple mathematical reasoning benchmarks, incurring only $\sim$3\% of the computational cost. When applied to stronger post-trained checkpoints, DOTS surpasses SOTA models and generalizes to out-of-domain benchmarks without re-tuning. Code is available at https://github.com/chaohaoyuan/DoTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。