arXiv:2605.25443cs.CL2026-05被引 1

通过对比学习实现多领域知识共享,提升大模型推理能力。

Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

论文配图:Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models
图 1 · 摘自论文原文
  • 设计对比学习机制,区分跨域可迁移路径与错误路径
  • 在多个数据集上显著提升推理模型表现,部分超越单领域训练
  • 适合需要跨领域泛化的大模型优化研究者

后训练已显著增强大推理模型(LRMs)的推理能力,尤其依赖强化学习(如组相对策略优化,GRPO)。然而,现有多领域强化学习方法在跨域优化中常因干扰导致各领域表现不一致。现有研究多关注缓解跨域干扰,却忽视了知识共享的关键作用。为此,本文提出多领域对比策略优化(MCPO),通过分析轨迹间的结构关系,以对比方式促进跨域知识共享与本域知识巩固。对于特定提示,MCPO将其他领域的可迁移推理路径视为正例,错误轨迹视为负例,促使正例表示一致、负例分离,并对同一领域正确轨迹进行对齐,构建统一的表示空间。实验表明,MCPO在多个领域均有效提升推理能力,甚至在某些情况下优于单领域训练。代码已开源。

原文摘要 · Abstract (English)

Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.

大模型推理对比学习多领域优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。