arXiv:2605.27659cs.LGcs.AI2026-05

通过隐变量与动态策略调整,提升仿真到现实的强化学习安全性和效率。

Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

论文配图:Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment
图 1 · 摘自论文原文
  • 利用隐变量建模环境上下文,实现跨场景策略迁移。
  • 根据隐变量估计精度动态调节风险水平,早期部署更安全。
  • 适合自动驾驶等对安全性要求高的真实系统部署。

由于资源限制和公共安全顾虑,许多网络物理系统(如自动驾驶车辆)的深度强化学习代理通常在模拟器中训练。然而,在真实环境中部署时,由于不可避免的模拟到现实差距,性能下降或安全违规现象频发。现有零样本方法(如鲁棒安全强化学习和域随机化)虽能缓解该问题,但常导致性能下降或残留安全风险,尤其在遭遇未建模系统动态时。为此,本文提出一种新型强化学习框架,通过概率隐嵌入与动态策略适配,实现安全高效的策略迁移。该框架考虑不同环境上下文下的约束马尔可夫决策过程(CMDP),借助元强化学习中的隐式上下文变量,从模拟经验中推断环境的隐表示。同时引入分布强化学习形式,依据隐变量估计精度动态调整部署策略的风险水平,从而在部署初期保障安全,并通过快速策略适应加速应对模拟到现实的差距。

原文摘要 · Abstract (English)

Due to limited resources and public safety concerns, deep reinforcement learning (RL) agents for many cyber-physical systems (e.g., autonomous vehicles) are first trained in simulators. However, when deployed in real world environments, they often suffer from performance degradation or safety violations because of the inevitable Sim2Real gap. Existing zero-shot approaches, such as robust safe RL and domain randomization, mitigate this issue but typically at the cost of degraded performance or residual safety risks when experiencing unmodeled system dynamics. To address these limitations, we propose a novel reinforcement learning framework that enables safe and efficient policy transfer via probabilistic latent embeddings and dynamic policy adaptation. We consider a family of Constrained Markov Decision Processes (CMDPs) under different environment contexts. By leveraging latent context variable in meta-RL, the proposed framework infers the latent representation of the environment from simulated experiences. Furthermore, it incorporates a distributional RL formulation, which allows risk levels of the deployed policy to be adjusted dynamically, based on the estimation accuracy of the latent context variable. This strategy promotes safety at the early deployment stage and improves efficiency through fast policy adaptation under the Sim2Real gap.

强化学习模拟到现实安全迁移隐变量建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。