用区块链共享专家经验,提升多智能体强化学习的训练效率与安全性。
Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
- 通过区块链共享专家模型,指导新智能体探索环境。
- 在多个任务中训练速度更快,且能抵抗错误和恶意模型干扰。
- 适合需要安全协作的多智能体系统开发者使用。
多智能体深度强化学习(MDRL)在协作或竞争环境中学习复杂行为方面前景广阔,但面临样本效率低、维度灾难和环境探索难等挑战。现有联邦强化学习方法存在模型限制和恶意行为问题,奖励塑造则需大量工程投入且易陷入局部最优。本文提出一种基于区块链的多专家示范克隆(MEDC)框架,利用专家示范引导新智能体探索。通过联盟区块链实现模型共享,用户可上传训练好的模型作为专家模型供他人调用,所有操作由智能合约管理,并通过IPFS分发。该框架在多个应用场景中测试,对比联邦强化学习、奖励塑造及模仿学习辅助强化学习,结果表明其在学习速度和抵御故障及恶意模型方面表现更优。
原文摘要 · Abstract (English)
Multi-Agent Deep Reinforcement Learning (MDRL) is a promising research area in which agents learn complex behaviors in cooperative or competitive environments. However, MDRL comes with several challenges that hinder its usability, including sample efficiency, curse of dimensionality, and environment exploration. Recent works proposing Federated Reinforcement Learning (FRL) to tackle these issues suffer from problems related to model restrictions and maliciousness. Other proposals using reward shaping require considerable engineering and could lead to local optima. In this paper, we propose a novel Blockchain-assisted Multi-Expert Demonstration Cloning (MEDC) framework for MDRL. The proposed method utilizes expert demonstrations in guiding the learning of new MDRL agents, by suggesting exploration actions in the environment. A model sharing framework on Blockchain is designed to allow users to share their trained models, which can be allocated as expert models to requesting users to aid in training MDRL systems. A Consortium Blockchain is adopted to enable traceable and autonomous execution without the need for a single trusted entity. Smart Contracts are designed to manage users and models allocation, which are shared using IPFS. The proposed framework is tested on several applications, and is benchmarked against existing methods in FRL, Reward Shaping, and Imitation Learning-assisted RL. The results show the outperformance of the proposed framework in terms of learning speed and resiliency to faulty and malicious models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。