解决联邦离线强化学习中的策略污染问题，提升低质量数据下的性能。
FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification
- 服务器用Q函数集成聚合，抗干扰能力强；设备端通过零阶搜索和正则化修正策略。
- 在不同数据质量和异构性下，性能均优于主流基线，最高提升37%。
- 适合资源受限的物联网设备，兼顾隐私保护与计算效率。
在物联网系统中，联邦学习推动了在线强化学习的发展，实现无需共享原始数据的并行策略训练。然而，与真实环境在线交互存在风险且成本高，促使离线联邦强化学习（FRL）的发展，即本地设备从固定数据集学习。尽管前景广阔，离线FRL在低质量、异构数据下可能失效：离线强化学习易陷入局部最优，而在联邦设置中，单个设备的次优策略会污染聚合模型，即策略污染。本文提出FORLER，结合服务器端的Q-ensemble聚合与设备端的演员修正机制。服务器通过鲁棒聚合多个设备的Q函数，抑制策略污染，并将重计算任务移出资源受限硬件，保障隐私。设备端采用零阶搜索识别高Q值动作，并引入定制正则化项引导策略向这些动作收敛。此外，采用δ周期性策略进一步降低本地计算开销。理论分析提供了安全策略改进的性能保证。大量实验表明，无论在何种数据质量与异构性条件下，FORLER始终优于强基线方法。
原文摘要 · Abstract (English)
In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A $δ$-periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。