用每步后悔值加速强化学习，让模型更快学会正确决策。
StepScorer: Accelerating Reinforcement Learning with Step-wise Scoring and Psychological Regret Modeling
- 每步计算行动与最优选择的后悔值，生成密集反馈信号
- 在Lunar Lander上比PPO快36%达到稳定性能
- 适合延迟奖励场景，如机器人控制、金融交易
强化学习常因稀疏奖励信号导致收敛缓慢，尤其在反馈延迟或稀少的复杂环境中。本文提出心理后悔模型（PRM），通过在每个决策步骤后计算后悔信号来加速学习。该信号基于当前动作价值与最优动作期望价值之差，将稀疏奖励转化为密集反馈，实现步骤级评分框架。实验表明，在Lunar Lander等基准环境上，PRM性能稳定达到时间比传统近端策略优化（PPO）快约36%。结果表明，该方法在连续控制任务和延迟反馈环境中尤为有效，适用于机器人、金融及自适应教育等需快速策略调整的真实场景。该方法将人类反事实思维形式化为可计算的后悔信号，连接行为经济学与强化学习。
原文摘要 · Abstract (English)
Reinforcement learning algorithms often suffer from slow convergence due to sparse reward signals, particularly in complex environments where feedback is delayed or infrequent. This paper introduces the Psychological Regret Model (PRM), a novel approach that accelerates learning by incorporating regret-based feedback signals after each decision step. Rather than waiting for terminal rewards, PRM computes a regret signal based on the difference between the expected value of the optimal action and the value of the action taken in each state. This transforms sparse rewards into dense feedback signals through a step-wise scoring framework, enabling faster convergence. We demonstrate that PRM achieves stable performance approximately 36\% faster than traditional Proximal Policy Optimization (PPO) in benchmark environments such as Lunar Lander. Our results indicate that PRM is particularly effective in continuous control tasks and environments with delayed feedback, making it suitable for real-world applications such as robotics, finance, and adaptive education where rapid policy adaptation is critical. The approach formalizes human-inspired counterfactual thinking as a computable regret signal, bridging behavioral economics and reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。