行为克隆比强化学习更优，揭示了当前方法在肿瘤治疗中的关键缺陷。
Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment
- 用参考控制器作为最优策略基准，对比不同算法性能。
- 行为克隆100%实现持续治愈，而强化学习训练后全部失败。
- 揭示了评论器误导导致策略退化，适合医疗决策研究者阅读。
自适应给药需要在降低肿瘤负荷的同时避免过度毒性。现有方法通常以历史或启发式方案为基准，无法判断策略是否达到最优。本文研究一个三类群肿瘤控制的常微分方程模型，通过最优控制分析确定理想用药模式——由脉冲剂量与奇异弧段组成的双峰结构，并构建数值控制器作为近似最优策略的代理。在持续治愈标准（连续200天肿瘤载量低于5%）下，从零开始训练的Soft Actor-Critic (SAC) 永远无法实现治愈。而对参考控制器进行行为克隆（BC）可完全复现该策略（30次种子实验均100%持续治愈），但对克隆策略进行SAC微调后，所有熵系数下均失效；TD3与基于行为克隆正则化的SAC也表现相同。这一现象在乘性药代动力学噪声下依然存在。在治愈轨迹中，崩溃后的评论器在96%的状态下将崩溃策略的动作评价高于参考动作，且集中于维持阶段，导致策略陷入非治愈性自适应治疗平衡态。正是参考控制器的存在使这一失败变得可解释：若仅与启发式基准比较，微调后策略会显得表现良好，实则已失效。
原文摘要 · Abstract (English)
Adaptive dosing requires policies that reduce tumor burden without excessive toxicity. Learned dosing policies are typically judged against historical or heuristic comparators, which cannot show whether a policy has found the best behavior available. We instead study a three-population tumor-control ODE in which optimal-control analysis fixes the form of a good schedule -- bang-bang dosing punctuated by a singular arc -- and construct a numerical controller of that form as a proxy for near-optimal behavior. Judged against this reference under a sustained-cure criterion -- 200 consecutive days below 5% carrying capacity -- Soft Actor-Critic (SAC) trained from scratch never reaches cure. Behavioral cloning (BC) of the reference reproduces it (100% sustained cure, 30/30 seeds), but SAC fine-tuning of the cloned policy destroys it across five entropy coefficients, and TD3 and BC-regularized SAC fail identically; the pattern persists under multiplicative pharmacokinetic action noise. Along curative trajectories the post-collapse critic ranks the collapsed-policy action above the reference action in 96% of states, concentrated in the maintenance phase, and the policy settles into a non-curative adaptive-therapy equilibrium. The reference is what makes this legible: against a heuristic comparator the fine-tuned policy would read as a competent controller rather than a failure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。