通过最大化轨迹总相关性，让智能体行为更简单、更鲁棒。
Maximum Total Correlation Reinforcement Learning
- 在强化学习中引入总相关性最大化，促使策略生成周期性、可压缩的轨迹。
- 在模拟机器人环境中，相比基线方法提升抗噪声和动态变化能力。
- 适合关注泛化性与鲁棒性的强化学习研究者。
简洁是一种强大的归纳偏置。在强化学习中，正则化用于简化策略，数据增强用于简化表征，稀疏奖励函数用于简化目标，其核心动机是通过聚焦本质来提升泛化性和鲁棒性。本文补充这一思路，提出在整段轨迹中最大化总相关性，以促进更简单的行为模式。我们设计了一种实用算法，基于下界近似优化所有模型（包括策略和状态表征）。在模拟机器人环境中，该方法自然生成周期性且可压缩的轨迹，相比基线方法在抗噪声和动态变化方面表现更优，同时在原始任务上也取得更好性能。
原文摘要 · Abstract (English)
Simplicity is a powerful inductive bias. In reinforcement learning, regularization is used for simpler policies, data augmentation for simpler representations, and sparse reward functions for simpler objectives, all that, with the underlying motivation to increase generalizability and robustness by focusing on the essentials. Supplementary to these techniques, we investigate how to promote simple behavior throughout the episode. To that end, we introduce a modification of the reinforcement learning problem that additionally maximizes the total correlation within the induced trajectories. We propose a practical algorithm that optimizes all models, including policy and state representation, based on a lower-bound approximation. In simulated robot environments, our method naturally generates policies that induce periodic and compressible trajectories, and that exhibit superior robustness to noise and changes in dynamics compared to baseline methods, while also improving performance in the original tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。