用绝对误差替代平方误差，能更有效控制强化学习策略的次优差距。
Rectifying Regression in Reinforcement Learning
- 改用绝对误差作为损失函数，更贴近价值预测的真实目标。
- 基于交叉熵损失的算法在线性强化学习中表现优于传统平方误差方法。
- 适合关注策略优化精度与训练稳定性的强化学习研究者。
本文通过分析基于价值的强化学习方法中损失函数的内在预测目标，揭示了均方误差（MSE）在控制策略次优性方面存在局限。理论证明，平均绝对误差（MAE）是更优的预测目标。进一步发现，二元交叉熵和分类交叉熵损失与MAE目标更匹配，而平方损失与MSE目标一致。实验表明，在线性强化学习任务中，最小化交叉熵损失的算法性能优于基于平方损失的方法。
原文摘要 · Abstract (English)
This paper investigates the impact of the loss function in value-based methods for reinforcement learning through an analysis of underlying prediction objectives. We theoretically show that mean absolute error is a better prediction objective than the traditional mean squared error for controlling the learned policy's suboptimality gap. Furthermore, we present results that different loss functions are better aligned with these different regression objectives: binary and categorical cross-entropy losses with the mean absolute error and squared loss with the mean squared error. We then provide empirical evidence that algorithms minimizing these cross-entropy losses can outperform those based on the squared loss in linear reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。