用压缩图像特征的进化算法玩太空入侵者,效率更高。
Playing Atari Space Invaders with Sparse Cosine Optimized Policy Evolution
- 用离散余弦变换压缩游戏画面,保留关键视觉特征
- 输入维度从33600降至15625,减少53%数据量
- 适合想用进化算法高效训练游戏智能体的研究者
进化方法在多种领域已被证明有效,但在游戏玩法中面临挑战,因视频游戏状态空间过大。随着输入状态扩大,策略必须增大以学习时间模式,导致可训练参数呈指数增长,搜索空间急剧膨胀,严重拖慢进化算法收敛速度。为在保持原始信息意义的前提下缩小输入空间,本文提出稀疏余弦优化策略演化(SCOPE)。该方法利用离散余弦变换(DCT)作为伪注意力机制,将输入状态转换为系数矩阵,通过截断与稀疏化降低维度,保留原输入中能量最高的特征。实验表明,以CMA-ES为优化器时,使用SCOPE策略在Atari游戏《Space Invaders》中表现优于未处理输入的OpenAI-ES和HyperNEAT等进化方法,也优于DQN和A3C等简单强化学习方法。其核心在于将输入规模从33,600降低至15,625(减少53%),并采用双线性仿射映射将稀疏DCT系数转化为策略动作。
原文摘要 · Abstract (English)
Evolutionary approaches have previously been shown to be effective learning methods for a diverse set of domains. However, the domain of game-playing poses a particular challenge for evolutionary methods due to the inherently large state space of video games. As the size of the input state expands, the size of the policy must also increase in order to effectively learn the temporal patterns in the game space. Consequently, a larger policy must contain more trainable parameters, exponentially increasing the size of the search space. Any increase in search space is highly problematic for evolutionary methods, as increasing the number of trainable parameters is inversely correlated with convergence speed. To reduce the size of the input space while maintaining a meaningful representation of the original space, we introduce Sparse Cosine Optimized Policy Evolution (SCOPE). SCOPE utilizes the Discrete Cosine Transform (DCT) as a pseudo attention mechanism, transforming an input state into a coefficient matrix. By truncating and applying sparsification to this matrix, we reduce the dimensionality of the input space while retaining the highest energy features of the original input. We demonstrate the effectiveness of SCOPE as the policy for the Atari game Space Invaders. In this task, SCOPE with CMA-ES outperforms evolutionary methods that consider an unmodified input state, such as OpenAI-ES and HyperNEAT. SCOPE also outperforms simple reinforcement learning methods, such as DQN and A3C. SCOPE achieves this result through reducing the input size by 53% from 33,600 to 15,625 then using a bilinear affine mapping of sparse DCT coefficients to policy actions learned by the CMA-ES algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。