用带记忆的强化学习优化化疗,解决医生看不清患者状态的难题
Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability

- 采用带LSTM的循环策略,让模型记住过去治疗信息
- 在观测不全时,肿瘤控制更稳定,正常细胞保护更好
- 适合临床中患者状态难以完全观测的场景
化疗剂量优化可建模为动态治疗方案,需在不确定性下做出序列决策,平衡肿瘤抑制与毒性。然而,多数强化学习方法假设能完整观测患者状态,这在临床实践中极少成立。本文研究记忆增强策略在部分可观测性下的化疗控制效果。采用基于TD3的循环方法,使用独立LSTM结构的演员-评论家网络,在DTR-Bench提供的AhnChemoEnv基准上评估了离策略与在线策略循环架构,对比前馈TD3与软演员-评论家算法。药代动力学和药效学变异性保持固定,以隔离隐藏状态不确定性和观测噪声,避免个体间差异干扰。在十组随机种子下,循环结构在完全可观测时仅有小幅提升,但在部分可观测时表现显著更优且更稳定,肿瘤抑制更一致,正常细胞保存效果更好。结果表明,当临床状态信息不完整或含噪时,基于记忆的策略尤为有效。
原文摘要 · Abstract (English)
Chemotherapy dose optimization can be formulated as a dynamic treatment regime, requiring sequential decisions under uncertainty that must balance tumor suppression against toxicity. However, most reinforcement learning approaches assume full observability of the patient state, a condition rarely met in clinical practice. We investigate whether memory-augmented policies can improve chemotherapy control under partial observability. To this end, we employ a recurrent TD3-based approach with separate LSTM actor-critic networks and evaluate it on the AhnChemoEnv benchmark from DTR-Bench, considering both off-policy and on-policy recurrent architectures against feed-forward TD3 and Soft Actor-Critic. Pharmacokinetic and pharmacodynamic variability are held fixed to isolate hidden-state uncertainty and observation noise and to avoid confounding effects from inter-patient variability. Across ten random seeds, recurrence yields modest benefit under full observability but substantially stronger and more stable performance under partial observability, with more consistent tumor suppression and improved normal-cell preservation. These findings indicate that memory-based policies are particularly beneficial when clinically relevant state information is incomplete or noisy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。