arXiv:2606.26574cs.LG2026-06

对比220种配置,发现分层策略在复杂动作空间中表现更优。

Revisiting Action Factorization for Complex Action Spaces

论文配图:Revisiting Action Factorization for Complex Action Spaces
图 1 · 摘自论文原文
  • 系统测试多种动作分解方法在三种算法和三类动作空间中的表现
  • 分层式双重架构在性能与计算开销间平衡最佳,自回归动作表现最强
  • 新构建轻量级环境,适合研究状态依赖的交互机制,适合强化学习研究者

许多现实世界的控制问题涉及混合离散-连续动作空间,如自动驾驶中的转向与变道、机器人或游戏中的瞄准与射击。尽管现有强化学习框架(如Gymnasium、PettingZoo、TorchRL、SeedRL、Mujoco等)支持复杂动作空间,但默认环境多为统一动作配置(如LunarLander、Walker2D、Cheetah、SMAC、SUMO、Ant、Atari)。标志性混合动作基准(如RoboCup 2D HFO、SC2LE、Platform、CARLA等)多为重型或存档实现,仅针对特定控制任务测试少数分解方法。本文对分层方法(独立网络、共享编码器、VDN、QPLEX、Joint、自回归)在三种算法(PPO、SAC、DQN)下,跨三类动作空间(离散化、混合、连续)在四个轻量级环境(Platform、hybrid-LunarLander、Hybrid-Shoot、CoopPush)中进行横断面分析,共220种配置。我们新增两个符合Gymnasium与PettingZoo规范的C++并行环境(CoopPush、Hybrid-Shoot),用于隔离状态依赖的交互挑战。引入VDN-PPO与PPO-MIX,采用分支批评器分配多头PPO的信用。这些变体优于所有测试的PPO分解方法。结果表明,分层双重架构在计算与性能间取得最优平衡,自回归动作整体表现最佳,原生连续SAC性能领先,但计算成本更高。

原文摘要 · Abstract (English)

Many real-world control problems involve hybrid discrete-continuous action spaces. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video-games. Despite real-world hybrid factorization and reinforcement learning framework support for complex action spaces (e.g., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc), the default environments within those frameworks often implement uniform action space configurations (LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari). Landmark hybrid-action benchmarks (RoboCup 2D HFO, SC2LE, Platform, CARLA, etc) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control. This article provides a cross-sectional study of factorization methods [independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive] on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]. Accounting for some invalid pairings such as joint-continuous, we are left with 220 configurations to analyze each method. We provide two new C++ parallel gymnasium and petting-zoo compliant environments [CoopPush, Hybrid-Shoot] to isolate particular challenges such as state-dependent inter-action dependence. Finally, we introduce VDN-PPO and PPO-MIX which use a branching critic to assign credit to multi-headed PPO. These variants out-perform all other tested PPO factorizations. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.

强化学习动作分解多智能体算法对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。