arXiv:2606.18521cs.LGcs.AI2026-06KDD

RLVR模型参数稀疏导致合并困难,提出新方法实现高效推理能力融合。

Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

论文配图:Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
图 1 · 摘自论文原文
  • 发现RLVR更新在参数空间中分布稀疏且近正交,难以直接合并
  • 新方法SAR-Merging通过敏感度仲裁与稀疏化修复合并损伤
  • 在数学与编程任务上显著优于现有方法,适合多能力融合场景

强化学习可验证奖励(RLVR)作为一种强大的后训练范式,能更好激发推理能力并抵抗灾难性遗忘。近期研究发现,相比监督微调(SFT),RLVR导致的参数更新高度稀疏且偏离主方向。这引发疑问:这种稀疏性是否使RLVR模型更易合并?然而我们发现相反结果——存在‘稀疏诅咒’:稀疏的更新在参数空间中分布更分散,形成近正交的捷径,导致聚合天然脆弱。这可能源于强化学习优化的随机性及涌现推理模式的多样性。不同于收敛于共享平坦盆地的SFT模型,标准合并方法会使RLVR模型严重退化。通过系统分析更新几何结构,我们揭示了失败机制,并提出针对RLVR参数空间特性的敏感度感知修复合并(SAR-Merging)。该方法基于费舍尔信息进行敏感度仲裁,再经幅度感知稀疏化与重缩放,以保护脆弱的推理路径。在数学与编码基准测试中,SAR-Merging显著优于现有方法,实现单任务提升与多能力融合。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting. Recent studies further reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT. This naturally raises the question: does such sparsity make RLVR models more amenable to model merging? If so, model merging would offer a scalable, training-free path to aggregate diverse reasoning capabilities from independently trained RLVR models. Surprisingly, we find the opposite, uncovering a sparsity curse: the sparse RLVR updates are spread farther apart in parameter space, forming near-orthogonal shortcuts that make aggregation inherently fragile. This is likely rooted in the stochasticity of RL optimization and the diversity of emergent reasoning patterns. Unlike SFT models that converge to shared, flat basins and merge naturally, RLVR models suffer severe degradation under standard merging methods. Through systematic empirical analysis of the update geometry, we characterize the mechanisms behind this failure and propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces. SAR-Merging resolves conflicts in overlapping update regions via Fisher Information-based sensitivity arbitration, followed by magnitude-aware sparsification and rescaling to preserve fragile reasoning pathways. Experiments on mathematical and coding benchmarks demonstrate that SAR-Merging substantially outperforms existing merging methods on RLVR models, enabling both single-task enhancement and multi-capability fusion.

模型合并强化学习稀疏更新推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。