提出多时尺度干预框架,高效管理资源并实现持久影响。
Learning Multi-Timescale Interventions under Safety and Resource Constraints
- 用增强状态追踪干预的累积与衰减效应,保持马尔可夫性。
- 在三类任务中平均表现最优,糖尿病模拟中提升21.5个百分点。
- 适合需长期干预且资源受限的医疗、工业控制场景。
许多序列决策问题中,干预方式具有不同时间尺度:某些即时生效,而另一些则产生持续影响,长期塑造未来状态。智能体需同时决定何时干预、选择何种时间模式及强度,还需考虑残留效应和有限干预资源。本文提出MINT:多时尺度干预网络训练方法。通过增强的干预状态来累积并衰减持久效应,结构化策略将干预模式选择与条件控制分离。不同于扩展策略执行的时序抽象,持久效应仍属于环境动态,可能与后续干预重叠。我们证明增强状态是干预历史的充分统计量,维持马尔可夫性,并建立了表格式Q-learning的贝尔曼收缩与几乎必然收敛。在持续控制的MuJoCo行走、随机库存管理及生理型1型糖尿病(T1DM)模拟器中,MINT在两个基准上取得最优平均主指标,且干预激活次数少于相同增强的扁平策略。其回报优势在所有约束预算下均存在,仅在无约束参考设置下趋近持平。在T1DM中实现$90.9\pm0.9\%$的血糖在目标范围内时间,且无低于范围时间,相比最强基线提升$21.5\%$点。代码、基准与所有实验配置见https://github.com/heywanrong/mint-rl。
原文摘要 · Abstract (English)
Many sequential decision problems offer qualitatively different ways of influencing the environment: some interventions act immediately, whereas others induce persistent effects that continue to shape future states long after the decision that initiated them. An agent must then decide jointly when to intervene, which temporal mode to use and how strongly, while accounting for residual effects and limited intervention resources. We introduce MINT: Multi-timescale Intervention Network Training. Persistent effects are carried by an augmented intervention state that accumulates and decays, while a structured policy separates intervention-mode selection from conditional control. Unlike temporal abstractions that extend policy execution, persistent-effect interventions remain part of the environment dynamics and may overlap with later interventions. We show that the augmented state is a sufficient statistic for the intervention history, preserving the Markov property, and establish Bellman contraction and almost-sure convergence of a tabular Q-learning instance. Across persistent-control MuJoCo locomotion, stochastic inventory management, and a physiologically grounded Type 1 Diabetes Mellitus (T1DM) simulator, MINT attains the best mean primary metric on two of three benchmarks while using fewer intervention activations than an identically augmented flat policy. Its return advantage is present at every binding intervention budget and narrows to parity in the unconstrained reference setting. In T1DM it achieves $90.9\pm0.9\%$ time in range with zero time below range, improving on the strongest baseline by $21.5\%$ points. Code, benchmarks and the configuration for every reported run are available at https://github.com/heywanrong/mint-rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。