用语义分割降低3D强化学习内存占用并提升性能
Enhancing Reinforcement Learning in 3D Environments through Semantic Segmentation: A Case Study in ViZDoom
- 用语义分割图像作为输入，减少记忆缓冲区占用
- 语义分割使内存消耗降低66.6%至98.6%
- 适合研究3D环境强化学习与视觉表征的开发者
在高维感官输入的3D环境中进行强化学习面临两大挑战：（1）为稳定学习所需的记忆缓冲区导致高内存消耗；（2）部分可观测马尔可夫决策过程（POMDP）的学习复杂性。本研究提出两种新型输入表示：仅语义分割（SS-only）和彩色图像+语义分割（RGB+SS），均在RGB图像上应用语义分割。实验在ViZDoom的死亡竞赛模式中进行，使用完美分割结果以控制评估。结果表明，SS-only可将记忆缓冲区内存消耗至少降低66.6%，在采用无损压缩技术（如游程编码）时最高降低98.6%。同时，RGB+SS显著提升了强化学习代理的性能，得益于额外的语义信息。此外，我们探索了基于密度的热力图作为可视化工具，用于分析代理移动模式并评估数据采集适用性。与先前方法的简要对比显示，本方法克服了在ViZDoom等3D环境中应用语义分割的常见问题。
原文摘要 · Abstract (English)
Reinforcement learning (RL) in 3D environments with high-dimensional sensory input poses two major challenges: (1) the high memory consumption induced by memory buffers required to stabilise learning, and (2) the complexity of learning in partially observable Markov Decision Processes (POMDPs). This project addresses these challenges by proposing two novel input representations: SS-only and RGB+SS, both employing semantic segmentation on RGB colour images. Experiments were conducted in deathmatches of ViZDoom, utilizing perfect segmentation results for controlled evaluation. Our results showed that SS-only was able to reduce the memory consumption of memory buffers by at least 66.6%, and up to 98.6% when a vectorisable lossless compression technique with minimal overhead such as run-length encoding is applied. Meanwhile, RGB+SS significantly enhances RL agents' performance with the additional semantic information provided. Furthermore, we explored density-based heatmapping as a tool to visualise RL agents' movement patterns and evaluate their suitability for data collection. A brief comparison with a previous approach highlights how our method overcame common pitfalls in applying semantic segmentation in 3D environments like ViZDoom.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。