用强化学习优化间歇通信下的控制策略,降低通信成本同时提升性能。
InterQ: A DQN Framework for Optimal Intermittent Control
- 设计双决策者框架:调度器间歇传状态,控制器基于不完整信息决策。
- 提出InterQ算法,神经网络逼近最优调度策略,显著优于周期与事件触发基线。
- 适用于资源受限的远程控制场景,如工业物联网与无人系统。
本文通过强化学习研究离散时间随机线性系统的通信-控制协同设计。考虑一个包含调度器和控制器的闭环系统:调度器持续监测系统状态,但仅间歇性地将状态发送给控制器以平衡通信开销与控制性能;控制器则基于接收到的不完整信息决定控制输入。在部分嵌套信息结构下,我们证明最优控制策略具有确定性等价形式,并分析了调度策略的定性行为。为求解最优调度策略,提出InterQ方法,采用深度神经网络近似Q函数。通过大量数值实验,分析调度策略空间,并与两种基线策略对比:(a) 多周期周期调度策略,(b) 事件触发策略。结果表明,所提方法性能全面优于两者。开源代码见 https://github.com/AC-sh/InterQ。
原文摘要 · Abstract (English)
In this letter, we explore the communication-control co-design of discrete-time stochastic linear systems through reinforcement learning. Specifically, we examine a closed-loop system involving two sequential decision-makers: a scheduler and a controller. The scheduler continuously monitors the system's state but transmits it to the controller intermittently to balance the communication cost and control performance. The controller, in turn, determines the control input based on the intermittently received information. Given the partially nested information structure, we show that the optimal control policy follows a certainty-equivalence form. Subsequently, we analyze the qualitative behavior of the scheduling policy. To develop the optimal scheduling policy, we propose InterQ, a deep reinforcement learning algorithm which uses a deep neural network to approximate the Q-function. Through extensive numerical evaluations, we analyze the scheduling landscape and further compare our approach against two baseline strategies: (a) a multi-period periodic scheduling policy, and (b) an event-triggered policy. The results demonstrate that our proposed method outperforms both baselines. The open source implementation can be found at https://github.com/AC-sh/InterQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。