arXiv:2410.14916cs.MAcs.AI2024-10中稿 · ACM Journal on Aut…被引 15

让多个智能体在导航中既高效又公平,避免某些个体负担过重。

Cooperation and Fairness in Multi-Agent Reinforcement Learning

  • 用最小化最大距离差异和公平性奖励来训练智能体
  • 效率提升14%,公平性提升5%(相比随机分配)
  • 无需为特定队形调整模型,通用性强

多智能体系统通常以最大化共享成本目标为目标,反映系统整体效率。但在交通与移动系统的资源受限环境中,效率可能以牺牲公平性为代价——某些智能体承担显著更高的成本或更低的回报。任务分配不均可能导致部分智能体获得不公平优势,而其他智能体承受过高代价。因此需要权衡效率与公平性。本文研究基于多智能体强化学习(MARL)的公平多智能体导航问题,将不同智能体行进距离的变异系数倒数作为公平性度量,探究智能体是否能在不显著牺牲效率的前提下学会公平行为(即不增加总行进距离)。实验表明,通过采用最小-最大公平距离目标分配并引入鼓励公平性的奖励项,智能体能够:(1)学习公平的目标分配;(2)仅依赖局部观测实现近乎完美的目标覆盖。在目标覆盖场景中,平均相较随机分配基线提升14%效率、5%公平性;相较于最优效率分配模型,公平性平均提升21%,效率仅下降7%。最后,该方法扩展至需按指定队形完成覆盖任务的环境,且无需针对特定队形定制模型。

原文摘要 · Abstract (English)

Multi-agent systems are trained to maximize shared cost objectives, which typically reflect system-level efficiency. However, in the resource-constrained environments of mobility and transportation systems, efficiency may be achieved at the expense of fairness -- certain agents may incur significantly greater costs or lower rewards compared to others. Tasks could be distributed inequitably, leading to some agents receiving an unfair advantage while others incur disproportionately high costs. It is important to consider the tradeoffs between efficiency and fairness. We consider the problem of fair multi-agent navigation for a group of decentralized agents using multi-agent reinforcement learning (MARL). We consider the reciprocal of the coefficient of variation of the distances traveled by different agents as a measure of fairness and investigate whether agents can learn to be fair without significantly sacrificing efficiency (i.e., increasing the total distance traveled). We find that by training agents using min-max fair distance goal assignments along with a reward term that incentivizes fairness as they move towards their goals, the agents (1) learn a fair assignment of goals and (2) achieve almost perfect goal coverage in navigation scenarios using only local observations. For goal coverage scenarios, we find that, on average, our model yields a 14% improvement in efficiency and a 5% improvement in fairness over a baseline trained using random assignments. Furthermore, an average of 21% improvement in fairness can be achieved compared to a model trained on optimally efficient assignments; this increase in fairness comes at the expense of only a 7% decrease in efficiency. Finally, we extend our method to environments in which agents must complete coverage tasks in prescribed formations and show that it is possible to do so without tailoring the models to specific formation shapes.

多智能体公平性强化学习导航

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。