区分图像强化学习中识别与决策的错误来源，提升模型泛化能力。
Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning
- 提出识别后悔与决策后悔概念，拆解图像强化学习中的误差
- 发现过冗余特征导致决策失误，过简特征导致识别偏差
- 在迷宫和Pong游戏中验证两种过拟合现象，适合研究泛化问题者
在基于图像的强化学习中，策略通常分为两步：先从原始图像提取低维特征（识别步骤），再基于提取的特征做出动作（决策步骤）。若特征提取出与性能虚假相关或对决策无关的特征，会导致泛化性能下降，称为观测过拟合。此时难以判断错误来自特征提取不佳还是决策能力不足。为此，本文引入识别后悔与决策后悔的概念，厘清观测过拟合的两类成因：过冗余表示（包含非必要特征，引发高决策后悔）与过简表示（仅保留训练中偶然相关的有限特征，引发高识别后悔）。最后，在迷宫环境和Atari游戏Pong中展示了两种过拟合的实例。
原文摘要 · Abstract (English)
In image-based reinforcement learning (RL), policies usually operate in two steps: first extracting lower-dimensional features from raw images (the "recognition" step), and then taking actions based on the extracted features (the "decision" step). Extracting features that are spuriously correlated with performance or irrelevant for decision-making can lead to poor generalization performance, known as observational overfitting in image-based RL. In such cases, it can be hard to quantify how much of the error can be attributed to poor feature extraction vs. poor decision-making. To disentangle the two sources of error, we introduce the notions of recognition regret and decision regret. Using these notions, we characterize and disambiguate the two distinct causes behind observational overfitting: over-specific representations, which include features that are not needed for optimal decision-making (leading to high decision regret), vs. under-specific representations, which only include a limited set of features that were spuriously correlated with performance during training (leading to high recognition regret). Finally, we provide illustrative examples of observational overfitting due to both over-specific and under-specific representations in maze environments and the Atari game Pong.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。