为扩散策略引入可校准的风险控制,让生成更安全可靠。
LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies
- 将每步去噪视为统计假设检验,用对数似然比控制生成方向。
- 在D4RL MuJoCo任务上提升返回值与分布外性能的平衡,满足用户设定风险阈值。
- 无需重训练,可直接插到现有扩散策略中使用,适合追求安全性的研究者。
扩散策略在离线强化学习中表现优异,但采样时通常依赖缺乏风险统计意义的启发式引导。本文提出LRT-Diffusion,一种基于风险感知的采样规则,将每个去噪步骤建模为无条件先验与状态条件策略头之间的序列假设检验。具体而言,累积对数似然比,并通过逻辑控制器门控条件均值,其阈值τ在零假设下一次性校准,以满足用户指定的Ⅰ类错误率α。这使引导从固定推动转变为基于证据的调整,具备用户可理解的风险预算。重要的是,训练阶段保持原样(双头结构,标准ε预测),仅在推理时引入该方法。LRT引导可自然与Q梯度结合:批评家梯度可在无条件均值、门控均值或二者混合处更新,形成从激进到保守的连续谱。我们在训练和测试时统一标准化状态与动作,并报告状态条件下的分布外(OOD)指标及回报。在D4RL MuJoCo任务上,相比强基准的Q引导方法,我们的实现提升了回报与OOD性能的权衡,同时严格满足预设α。理论上,我们建立了α水平校准、简洁的稳定性界,并给出回报对比,揭示当支持外误差占主导时LRT优于Q引导的情形。总体而言,LRT-Diffusion是一种即插即用的推理阶段方法,为离线强化学习中的扩散策略提供了原则性、可校准的风险控制。
原文摘要 · Abstract (English)
Diffusion policies are competitive for offline reinforcement learning (RL) but are typically guided at sampling time by heuristics that lack a statistical notion of risk. We introduce LRT-Diffusion, a risk-aware sampling rule that treats each denoising step as a sequential hypothesis test between the unconditional prior and the state-conditional policy head. Concretely, we accumulate a log-likelihood ratio and gate the conditional mean with a logistic controller whose threshold tau is calibrated once under H0 to meet a user-specified Type-I level alpha. This turns guidance from a fixed push into an evidence-driven adjustment with a user-interpretable risk budget. Importantly, we deliberately leave training vanilla (two heads with standard epsilon-prediction) under the structure of DDPM. LRT guidance composes naturally with Q-gradients: critic-gradient updates can be taken at the unconditional mean, at the LRT-gated mean, or a blend, exposing a continuum from exploitation to conservatism. We standardize states and actions consistently at train and test time and report a state-conditional out-of-distribution (OOD) metric alongside return. On D4RL MuJoCo tasks, LRT-Diffusion improves the return-OOD trade-off over strong Q-guided baselines in our implementation while honoring the desired alpha. Theoretically, we establish level-alpha calibration, concise stability bounds, and a return comparison showing when LRT surpasses Q-guidance-especially when off-support errors dominate. Overall, LRT-Diffusion is a drop-in, inference-time method that adds principled, calibrated risk control to diffusion policies for offline RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。