通过语义聚类提升深度强化学习的可解释性
Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
- 引入新型语义聚类模块，结合降维与在线聚类
- 实验证明可揭示DRL内部的语义分组特征
- 适合关注模型决策逻辑与结构分析的研究者
本文探索深度强化学习（DRL）中语义聚类特性，以提升其可解释性并深化对内部语义结构的理解。语义聚类指神经网络在特征空间中根据输入的语义相似性进行分组的能力。我们提出一种新型DRL架构，集成一个结合特征降维与在线聚类的语义聚类模块，可无缝嵌入训练流程，克服t-SNE不稳定性问题，并避免传统语义分析方法所需的大量人工标注。实验验证了该模块的有效性，成功揭示了DRL中的语义聚类特性。此外，我们基于这些特性提出新的分析方法，为策略的层次结构及特征空间中的语义组织提供洞察。代码已开源：https://github.com/ualiangzhang/semantic_rl。
原文摘要 · Abstract (English)
In this paper, we explore semantic clustering properties of deep reinforcement learning (DRL) to improve its interpretability and deepen our understanding of its internal semantic organization. In this context, semantic clustering refers to the ability of neural networks to cluster inputs based on their semantic similarity in the feature space. We propose a DRL architecture that incorporates a novel semantic clustering module that combines feature dimensionality reduction with online clustering. This module integrates seamlessly into the DRL training pipeline, addressing the instability of t-SNE and eliminating the need for extensive manual annotation inherent to prior semantic analysis methods. We experimentally validate the effectiveness of the proposed module and demonstrate its ability to reveal semantic clustering properties within DRL. Furthermore, we introduce new analytical methods based on these properties to provide insights into the hierarchical structure of policies and semantic organization within the feature space. Our code is available at https://github.com/ualiangzhang/semantic_rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。