提出可应对高风险的多模态离线强化学习框架，兼顾安全与表现。
RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
- 用扩散模型做生成策略，结合分布值函数与条件风险价值优化
- 在Stochastic-D4RL上实现0.1分位数风险收益提升，同时保持高回报
- 行为克隆约束降低分布外动作，避免灾难性失败，适合高危场景
在无法进行在线数据收集的安全关键领域，离线强化学习若不能避免灾难性低尾风险，则难以应用。已有风险规避方法要么依赖价值/模型悲观性，要么限制策略表达能力；而表达能力强的扩散/流模型多用于无风险场景。本文提出风险感知的多模态演员-评论家（RAMAC）框架，采用生成式演员（如扩散/流模型）与分布式评论家，联合优化条件风险价值（CVaR）与行为克隆（BC），实现复杂多模态环境下的风险敏感学习。分析表明，通过行为克隆控制策略偏离，能有效抑制分布外动作，稳定CVaR。以扩散演员实例化后，在2维风险博弈和Stochastic-D4RL上均实现$\mathrm{CVaR}_{0.1}$持续提升，且保持优异回报。代码与结果见项目主页。
原文摘要 · Abstract (English)
In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) is attractive only if policies achieve high returns without catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of either (i) value/model-based pessimism or (ii) restricted policy classes that limit expressiveness, whereas diffusion/flow-based expressive generative policies have largely been used in risk-neutral settings. We introduce \textbf{Risk-Aware Multimodal Actor-Critic (RAMAC)}, a simple, modular, model-free framework that couples an expressive generative actor (e.g., diffusion/flow) with a distributional critic and optimizes a composite objective that combines Conditional Value-at-Risk (CVaR) with behavioral cloning (BC), enabling risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further provide an objective-level analysis showing that controlling behavior divergence via BC suppresses OOD actions and stabilizes CVaR. Instantiating RAMAC with a diffusion actor, we illustrate these insights on a 2-D risky bandit and evaluate on Stochastic-D4RL, observing consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns. The code and experimental results are available on the \href{https://kaifukazawa.github.io/ramac-project/} {project website}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。