arXiv:2602.08655cs.LG2026-02

用几何方法抑制离线强化学习中的异常动作,提升医疗决策安全性和稳定性。

From Robotics to Sepsis Treatment: Offline RL via Geometric Pessimism

  • 基于状态-动作嵌入空间的近邻距离设计惩罚项,实现高效保守性约束。
  • 在D4RL基准上比标准IQL高出18分以上,种子间波动降低4倍。
  • 适用于医疗等高风险场景,能避免行为克隆,保持临床可解释性。

离线强化学习有望从静态数据中恢复最优策略,但仍易高估分布外动作,尤其在断裂稀疏的数据流形上。现有方法需在计算效率与性能间权衡:如CQL虽严格保守但耗算力巨大,而高效的方法如IQL在病态数据集上常失效并退化为行为克隆。本文提出几何悲观(Geometric Pessimism),一种模块化、计算高效的框架,通过预计算状态-动作对在嵌入空间中的k近邻距离来构建密度惩罚,以奖励塑形方式注入分布外保守性,训练开销仅为O(1)。在D4RL MuJoCo基准上,Geo-IQL在敏感不稳定的medium-replay任务中表现优于标准IQL超过18点,且种子间标准差降低4倍;在稳定流形上无性能下降。更重要的是,在MIMIC-III脓毒症危重症数据集上,标准IQL退化为行为克隆,而Geo-IQL实现主动策略优化,维持安全约束下达到86.4%的终期临床一致性,高于IQL的75%。结果表明,几何悲观为关键现实决策系统提供必要正则化,可安全跳出局部最优。

原文摘要 · Abstract (English)

Offline Reinforcement Learning (RL) promises the recovery of optimal policies from static datasets, yet it remains susceptible to the overestimation of out-of-distribution (OOD) actions, particularly in fractured and sparse data manifolds. Current solutions necessitate a trade-off between computational efficiency and performance. Methods like CQL offer rigorous conservatism but require tremendous compute power while efficient expectile-based methods like IQL often fail to correct OOD errors on pathological datasets, collapsing to Behavioural Cloning. In this work, we propose Geometric Pessimism, a modular, compute-efficient framework that augments standard IQL with density-based penalty derived from k-nearest-neighbour distances in the state-action embedding space. By pre-computing the penalties applied to each state-action pair, our method injects OOD conservatism via reward shaping with a O(1) training overhead to the training loop. Evaluated on the D4RL MuJoCo benchmark, our method, Geo-IQL outperforms standard IQL on sensitive and unstable medium-replay tasks by over 18 points, while reducing inter-seed standard-deviation by 4 times. Furthermore, Geo-IQL does not degrade performance on stable manifolds. Crucially, we validate our algorithm on the MIMIC-III Sepsis critical care dataset. While standard IQL collapses to behaviour cloning, Geo-IQL demonstrates active policy improvement. Maintaining safety constraints, it achieves 86.4% terminal agreement with clinicians compared to IQL's 75%. Our results suggest that geometric pessimism provides the necessary regularisation to safely overcome local optima in critical, real-world decision systems.

离线强化学习医疗决策几何正则化策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。