高效运行多智能体博弈的实时决策引擎,速度提升超3500倍。
A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations

- 用数据导向设计与缓存对齐优化,实现高吞吐环境执行。
- 单机16核下达3300万步/秒,十智能体仍保持700万步/秒。
- 适合需快速训练多智能体策略的强化学习研究者使用。
强化学习在去中心化部分可观测马尔可夫决策过程(Dec-POMDP)中样本效率低。为应对这一挑战,如SampleFactory、EnvPool、Brax和IsaacLab等项目将经典环境(如MuJoCo和Atari)的并行执行迁移到C++线程池或GPU以降低计算开销。本文聚焦于人-智能体协同操作中的决策层优化,提出一个原生用C++构建的高效Dec-POMDP引擎——HASE。通过采用数据导向设计(DOD)、显式64字节缓存行对齐消除虚假共享,并利用固定内存与直接内存访问(DMA)实现零拷贝的PyTorch内存桥接,该引擎在单智能体、1024个环境、去中心化观测设置下,于AMD Ryzen 9950X(16核)上实现了高达33,000,000步/秒(SPS)的吞吐量;当引入10个智能体时,吞吐量降至700万步/秒,其中随机动作生成占总运行时间约1/3。相比基线单线程向量化NumPy实现,性能提升约3,500倍。该引擎成功在数分钟内训练出基于PPO、DQN和SAC的协作多智能体策略,验证了其性能与通用性。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) algorithms exhibit high sample complexity, particularly when applied to Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). As a response, projects such as SampleFactory, EnvPool, Brax, and IsaacLab migrate parallel execution of classic environments such as MuJoCo and Atari into C++ thread pools or the GPU to decrease the computational cost of environment steps. We are interested in optimizing the decision-level of human-AI joint operations, so we introduce a compute-efficient Dec-POMDP engine natively architected in C++ called Hide-And-Seek-Engine. By employing Data-Oriented Design (DOD) principles, explicit 64-byte cache-line alignment to remove false sharing, and a zero-copy PyTorch memory bridge using pinned memory and Direct Memory Access (DMA), our engine sustains throughput of up to 33,000,000 steps per second (SPS) in a single-agent, 1024-environment, decentralized observations on an AMD Ryzen 9950X (16 cores). Ten agents reduces FPS to 7M SPS with generating random actions contributing 1/3rd the total runtime for reference. The engine achieves a throughput increase of approximately 3,500$\times$ over the baseline single threaded vectorized NumPy implementation and successfully trains cooperative multi-agent policies via PPO, DQN, and SAC in minutes, validating both its performance and generality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。