用在线动作堆叠让强化学习更贴近真实空管指令
Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control
- 训练时用5个简单动作，推理时自动组合成复杂指令
- 指令频率降低90%以上，性能媲美37维动作空间模型
- 适合空管、交通控制等需低频高精度指令的场景
我们提出在线动作堆叠（online action-stacking），一种强化学习策略的推理时封装方法，可在保持训练时小离散动作空间（仅5个）的同时，生成符合航空管制实际的复合指令。策略通过简单的航向或高度调整训练，并引入动作抑制惩罚以减少指令频率，使智能体以短时突发方式发出指令。推理阶段，该方法将这些原始动作序列合并为领域适配的复合放行指令。基于Proximal Policy Optimisation与BluebirdDT数字孪生平台，在横向导航、爬升下降至目标高度层及双机避撞等任务中验证，相比抑制基线，指令数量显著减少，且性能接近使用37维动作空间训练的策略。结果表明，该方法有效弥合了标准强化学习与空管操作需求之间的差距，为复杂控制场景的扩展提供了简单可行的机制。
原文摘要 · Abstract (English)
We introduce online action-stacking, an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。