arXiv:2604.27162cs.MAcs.LG2026-04

高效运行多智能体博弈的实时决策引擎,速度提升超3500倍。

A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations

论文配图:A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations
图 1 · 摘自论文原文
  • 用数据导向设计与缓存对齐优化,实现高吞吐环境执行。
  • 单机16核下达3300万步/秒,十智能体仍保持700万步/秒。
  • 适合需快速训练多智能体策略的强化学习研究者使用。

强化学习在去中心化部分可观测马尔可夫决策过程(Dec-POMDP)中样本效率低。为应对这一挑战,如SampleFactory、EnvPool、Brax和IsaacLab等项目将经典环境(如MuJoCo和Atari)的并行执行迁移到C++线程池或GPU以降低计算开销。本文聚焦于人-智能体协同操作中的决策层优化,提出一个原生用C++构建的高效Dec-POMDP引擎——HASE。通过采用数据导向设计(DOD)、显式64字节缓存行对齐消除虚假共享,并利用固定内存与直接内存访问(DMA)实现零拷贝的PyTorch内存桥接,该引擎在单智能体、1024个环境、去中心化观测设置下,于AMD Ryzen 9950X(16核)上实现了高达33,000,000步/秒(SPS)的吞吐量;当引入10个智能体时,吞吐量降至700万步/秒,其中随机动作生成占总运行时间约1/3。相比基线单线程向量化NumPy实现,性能提升约3,500倍。该引擎成功在数分钟内训练出基于PPO、DQN和SAC的协作多智能体策略,验证了其性能与通用性。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) algorithms exhibit high sample complexity, particularly when applied to Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). As a response, projects such as SampleFactory, EnvPool, Brax, and IsaacLab migrate parallel execution of classic environments such as MuJoCo and Atari into C++ thread pools or the GPU to decrease the computational cost of environment steps. We are interested in optimizing the decision-level of human-AI joint operations, so we introduce a compute-efficient Dec-POMDP engine natively architected in C++ called Hide-And-Seek-Engine. By employing Data-Oriented Design (DOD) principles, explicit 64-byte cache-line alignment to remove false sharing, and a zero-copy PyTorch memory bridge using pinned memory and Direct Memory Access (DMA), our engine sustains throughput of up to 33,000,000 steps per second (SPS) in a single-agent, 1024-environment, decentralized observations on an AMD Ryzen 9950X (16 cores). Ten agents reduces FPS to 7M SPS with generating random actions contributing 1/3rd the total runtime for reference. The engine achieves a throughput increase of approximately 3,500$\times$ over the baseline single threaded vectorized NumPy implementation and successfully trains cooperative multi-agent policies via PPO, DQN, and SAC in minutes, validating both its performance and generality.

强化学习多智能体高性能计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。