提出新算法抑制延迟伤害下的策略重复失效，提升强化学习安全性。
Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm
- 用环境级记忆记录历史伤害痕迹，动态调整动作转移概率
- 在50-1000节点图扩散任务中将重放大增益从0.98降至0.33
- 适合平台类系统中需长期安全约束的强化学习场景
强化学习中的安全通常通过目标函数设计实现，且假设环境状态转移是静态的。但在延迟伤害场景下，相同可观测状态-动作对重复出现会引发重复性危害（replay）。本文提出回放抑制诊断（RSD）协议，用于在固定策略下隔离此问题。研究发现，在可观测转移核不变的前提下，无法结构化抑制回放而不引起回放时刻动作分布的持续偏移。针对平台类系统，提出后悔感知策略优化（RAPO），在环境中引入持久伤害痕迹与伤疤场，并采用有界、质量守恒的转移重加权机制，降低历史有害区域的可达性。在50-1000节点的图扩散任务中，RAPO成功抑制回放，使250节点图上的重放大增益（RAG）从0.98降至0.33，同时保持82%的任务回报。仅在回放阶段禁用转移变形后，重放大增益回升至0.91，证实环境级变形是关键因果机制。
原文摘要 · Abstract (English)
Safety in reinforcement learning (RL) is typically enforced through objective shaping while keeping environment dynamics stationary with respect to observable state-action pairs. Under delayed harm, this can lead to replay: after a washout period, reintroducing the same stimulus under matched observable conditions reproduces a similar harmful cascade. We introduce the Replay Suppression Diagnostic (RSD), a controlled exposure-decay-replay protocol that isolates this failure mode under frozen-policy evaluation. We show that, under stationary observable transition kernels, replay cannot be structurally suppressed without inducing a persistent shift in replay-time action distributions. Motivated by platform-mediated systems, we propose Regret-Aware Policy Optimization (RAPO), which augments the environment with persistent harm-trace and scar fields and applies a bounded, mass-preserving transition reweighting to reduce reachability of historically harmful regions. On graph diffusion tasks (50-1000 nodes), RAPO suppresses replay, reducing re-amplification gain (RAG) from 0.98 to 0.33 on 250-node graphs while retaining 82\% of task return. Disabling transition deformation only during replay restores re-amplification (RAG 0.91), isolating environment-level deformation as the causal mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。