用小批量风险映射改进强化学习,让机器人导航更抗风险。
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study

- 用多步样本平均构造可估的风险映射,解决非线性依赖问题。
- 仅用2个样本就显著降低结果波动,且提升平均表现。
- 适合需要稳定决策的机器人任务,如水下巡检与信息采集。
我们研究马尔可夫决策过程中的控制问题,其中策略质量由动态、时间一致的马尔可夫风险度量评估,而非期望折扣成本。结合此类度量的主要障碍在于转移风险映射对转移核呈非线性依赖,无法从单次观测中估计。为此,我们采用小批量转移风险映射:将映射应用于N个独立的下一状态样本的经验分布,并取平均,该映射仍保持一致性。作为N个下一状态值函数的期望,它可被无偏单样本估计。我们将此映射嵌入双深度Q网络,分析两种估计偏差来源,提出适用于远超表格方法范围的状态空间的风险规避型Q学习方法。在水下机器人导航任务中应用:车辆需访问采集点收集随机信息载荷,并送达传输点,每一步均面临被摧毁的风险。通过层次分解,路径执行交由精确图搜索,学习仅限于‘采集或传输’的高层决策。基于问题对称性的低维特征映射替代原始状态编码。在300个未见环境上的实验表明,策略可泛化至训练中未出现的规模,且当仿真存在偏差时,仅N=2即能同时降低结果分布的上半方差并改善其均值——这体现了相干风险度量与分布鲁棒性之间的对偶关系。
原文摘要 · Abstract (English)
We study the control of Markov decision processes in which the quality of a policy is evaluated by a dynamic, time-consistent Markov risk measure rather than by an expected discounted cost. The main obstacle to combining such measures with reinforcement learning is that a transition risk mapping depends on the transition kernel in a nonlinear way, and therefore cannot be estimated from a single observed transition. We remove this obstacle by employing mini-batch transition risk mappings: the mapping is applied to the empirical measure of $N$ independent next-state samples, and the result is averaged. The resulting mapping is again coherent. However, as an expected value of a function of $N$ next-state values, it admits an unbiased one-sample estimator. We embed this mapping into a double deep Q-network, analyze the two sources of estimation bias that arise, and obtain a risk-averse Q-learning method applicable to state spaces far beyond the reach of tabular schemes. The method is applied to an underwater robot navigation problem, in which a vehicle must visit collection points, gather stochastic information payloads, and deliver them at transmission points, while exposed at each step to the risk of destruction. A hierarchical decomposition delegates path execution to an exact graph search and confines learning to the high-level ``collect or transmit'' decision. A low-dimensional feature map, invariant under the symmetries of the problem, replaces the raw state--configuration encoding. In experiments on $300$ held-out environments, the resulting policies transfer to instance sizes never seen in training, and already $N=2$ reduces the upper semideviation of the outcome distribution while simultaneously improving its mean whenever the simulator is misspecified---an empirical counterpart of the duality between coherent risk measures and distributional robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。