提出一种新方法，量化强化学习中观测的非马尔可夫性，帮助发现传感器噪声等问题。
Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learning
- 用随机森林和岭回归检测历史观测是否能提升预测精度
- 在6个环境、3种算法下验证，噪声越强，分数越高（最高0.78）
- 适合研究传感器问题或改进模型架构的研究者使用
强化学习假设观测满足马尔可夫性质，但现实传感器常因噪声相关性、延迟或部分可观测性违反该假设。现有性能指标将马尔可夫失效与其他低效因素混淆，缺乏检测工具。本文提出基于预测的马尔可夫违规评分（MVS），量化观测轨迹中的非马尔可夫结构。先用随机森林去除非线性马尔可夫兼容动态；再用岭回归检验历史观测是否在残差上降低预测误差，超越当前观测带来的收益。所得分数范围为[0,1]，无需因果图构建。评估涵盖六个环境（CartPole, Pendulum, Acrobot, HalfCheetah, Hopper, Walker2d）、三种算法（PPO, A2C, SAC）、六种强度的受控AR(1)噪声及每条件10个种子。后验检测显示，16组环境-算法中7组噪声强度与MVS呈显著正单调关系（斯皮尔曼秩相关最高0.78，重复测量分析确认）；训练时加噪下，16组中有13组出现显著奖励下降。在低维环境中发现反常现象：随机森林吸收噪声信号，导致真实违规增强时MVS反而下降，此失败模式被详细分析。实用性实验表明，MVS能正确识别部分可观测性并指导架构选择，完全恢复因非马尔可夫观测损失的性能。源码见https://github.com/NAVEENMN/Markovianes。
原文摘要 · Abstract (English)
Reinforcement learning algorithms assume that observations satisfy the Markov property, yet real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability. Standard performance metrics conflate Markov breakdowns with other sources of suboptimality, leaving practitioners without tools to detect such violations. This paper introduces a prediction-based Markov Violation Score (MVS) that quantifies non-Markovian structure in observation trajectories. A random forest first removes nonlinear Markov-compliant dynamics; ridge regression then tests whether historical observations reduce prediction error on the residuals beyond what the current observation provides. The resulting score is bounded in [0, 1] and requires no causal graph construction. Evaluation spans six environments (CartPole, Pendulum, Acrobot, HalfCheetah, Hopper, Walker2d), three algorithms (PPO, A2C, SAC), controlled AR(1) noise at six intensity levels, and 10 seeds per condition. In post-hoc detection, 7 of 16 environment-algorithm pairs, primarily high-dimensional locomotion tasks, show significant positive monotonicity between noise intensity and MVS (Spearman rho up to 0.78, confirmed under repeated-measures analysis); under training-time noise, 13 of 16 pairs exhibit statistically significant reward degradation. An inversion phenomenon is documented in low-dimensional environments where the random forest absorbs the noise signal, causing MVS to decrease as true violations grow, a failure mode analyzed in detail. A practical utility experiment demonstrates that MVS correctly identifies partial observability and guides architecture selection, fully recovering performance lost to non-Markovian observations. Source code to reproduce all results is available at https://github.com/NAVEENMN/Markovianes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。