异构智能体协作强化学习，让不同模型互相提升，推理时仍独立运行。
Heterogeneous Agent Collaborative Reinforcement Learning
- 异构智能体共享验证过的经验数据，训练时协作优化，推理时独立执行。
- 在多个推理基准上，相比双倍采样方法提升3.6%，但仅用一半采样成本。
- 适用于不同能力的模型协同，尤其适合大模型间知识互传的场景。
我们提出异构智能体协作强化学习（HACRL），一种新型可验证奖励强化学习问题，解决孤立多智能体策略优化效率低下的问题。HACRL允许异构智能体在训练中共享已验证的轨迹以相互提升，而在推理时独立运行。与基于大语言模型的多智能体强化学习不同，HACRL无需协调部署；与有/无偏置蒸馏不同，它支持异构智能体间的双向互学，而非单向同质师生传递。基于此，我们提出HACPO算法，通过原则性轨迹共享实现样本高效利用与跨智能体知识迁移。为缓解能力差异与策略分布偏移，HACPO引入四项定制机制，并保证无偏优势估计的理论保障。在多种异构模型组合与推理基准上的广泛实验表明，HACPO持续提升所有参与智能体的表现，平均比使用双倍轨迹的GSPO方法高3.6%，且仅需一半的轨迹开销。
原文摘要 · Abstract (English)
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。