解决异构奖励下个性化大模型的联邦强化学习难题
FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards
- 采用无评论家架构的GRPO，支持设备端高效训练
- 实现最高2.2%准确率提升，兼顾全局性能与个性化
- 适合隐私敏感场景下的多目标推理模型优化
组相对策略优化（GRPO）作为基于在线多目标强化学习提升大语言模型推理能力的有效方法受到关注。尽管在私有数据上的个性化需求日益增长，传统强化学习对齐因需维护独立评论家网络而难以在设备端进行联邦学习。GRPO无需评论家的设计使设备端训练成为可能，但迁移至联邦设置后面临异构奖励定义、多目标优化不平衡及高训练成本等系统性挑战。本文提出FedMOA，一种面向异构奖励的联邦GRPO框架。通过超梯度下降实现在线自适应加权机制，优先保障主要推理任务在辅助目标饱和时的稳定性；服务器端则采用任务与精度感知的聚合策略，优先融合高质量更新。在数学推理与代码生成基准上的实验表明，FedMOA持续优于联邦平均，准确率最高提升2.2%，同时改善全局性能、个性化程度与多目标平衡性。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has recently emerged as an effective approach for improving the reasoning capabilities of large language models through online multi-objective reinforcement learning. While personalization on private data is increasingly vital, traditional Reinforcement Learning (RL) alignment is often memory-prohibitive for on-device federated learning due to the overhead of maintaining a separate critic network. GRPO's critic-free architecture enables feasible on-device training, yet transitioning to a federated setting introduces systemic challenges: heterogeneous reward definitions, imbalanced multi-objective optimization, and high training costs. We propose FedMOA, a federated GRPO framework for multi-objective alignment under heterogeneous rewards. FedMOA stabilizes local training through an online adaptive weighting mechanism via hypergradient descent, which prioritizes primary reasoning as auxiliary objectives saturate. On the server side, it utilizes a task- and accuracy-aware aggregation strategy to prioritize high-quality updates. Experiments on mathematical reasoning and code generation benchmarks demonstrate that FedMOA consistently outperforms federated averaging, achieving accuracy gains of up to 2.2% while improving global performance, personalization, and multi-objective balance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。