测试时防护可让糖尿病强化学习模型在新患者上依然安全有效。
Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed
- 用训练好的动态模型过滤危险动作，实现部署时安全防护。
- 在8种算法、3类糖尿病、3个年龄组中提升13%-14%血糖达标时间。
- 适合关注医疗强化学习安全性的研究者和临床应用开发者。
安全强化学习（Safe RL）算法通常在固定训练条件下评估。本文以糖尿病管理为关键安全场景，研究训练期的安全保证能否在分布偏移下延续至部署阶段。我们在统一的临床模拟器上评测了多种安全强化学习算法，发现训练中满足约束的策略在未见过的患者上常违反安全要求，存在安全泛化差距。我们证明，测试时防护机制——通过学习的动力学模型过滤不安全动作——能有效恢复各类算法与患者群体的安全性。在8种安全强化学习算法、3种糖尿病类型及3个年龄组的实验中，该方法使强基线模型如PPO-Lag和CPO的血糖达标时间（Time-in-Range）提升13–14%，同时降低临床风险指数与葡萄糖波动性。所构建的模拟器与基准平台为安全关键控制领域中的分布偏移安全性研究提供了基础。代码已开源：https://github.com/safe-autonomy-lab/GlucoSim 与 https://github.com/safe-autonomy-lab/GlucoAlg。
原文摘要 · Abstract (English)
Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。