优化强化学习的马尔可夫决策过程设计，提升仿真到现实的迁移效果。
Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning
- 通过调整状态、奖励等五类MDP设计，系统研究其对仿真到现实迁移的影响。
- 基于物理的动力学模型在严格精度约束下实现50%真实成功率，简化模型则完全失败。
- 为工业过程控制中的强化学习部署提供可落地的建模指导，适合工程应用者参考。
强化学习在工业过程控制中展现出巨大潜力，但训练于仿真环境的策略在部署到真实硬件时常面临显著的仿真到现实差距。本文系统分析了马尔可夫决策过程（MDP）的核心设计选择——状态组成、目标包含、奖励设计、终止条件及环境动力学模型——对这一迁移性能的影响。基于颜色混合任务，在仿真与真实世界实验中评估了不同MDP配置和混合动态的表现。我们在真实硬件上验证了结果，表明基于物理的动力学模型在严格精度约束下可实现最高50%的真实成功率，而简化模型则完全失效。研究结果为工业过程控制中强化学习的部署提供了实用的MDP设计指南。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has demonstrated strong potential for industrial process control, yet policies trained in simulation often suffer from a significant sim-to-real gap when deployed on physical hardware. This work systematically analyzes how core Markov Decision Process (MDP) design choices -- state composition, target inclusion, reward formulation, termination criteria, and environment dynamics models -- affect this transfer. Using a color mixing task, we evaluate different MDP configurations and mixing dynamics across simulation and real-world experiments. We validate our findings on physical hardware, demonstrating that physics-based dynamics models achieve up to 50% real-world success under strict precision constraints where simplified models fail entirely. Our results provide practical MDP design guidelines for deploying RL in industrial process control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。