从稀疏轨迹标签中学习每一步的安全违规责任，提升强化学习安全性。
TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels
- 通过序列违规估计器分解每步违规概率，生成可优化的信用信号。
- 在多个连续控制任务中，违规满足率显著优于基线方法。
- 无需已知成本函数或阈值，适合安全约束不明确的场景。
在强化学习中，当安全约束隐含且无法密集测量时，确保安全行为极具挑战。许多场景下，仅能获得整个轨迹的粗粒度反馈（如是否超出未知安全阈值）。我们提出TraCeS（基于轨迹的安全约束估计），一种从稀疏轨迹级标签中学习每步违规信用的方法。TraCeS训练一个序列违规估计器，其每步信用信号基于轨迹尚未违规的概率，并将该信号整合进约束策略优化。该方法无需已知代价函数或阈值，且与标准连续控制算法兼容。我们提供了学习目标引入近似误差的理论分析，并在多个连续控制基准上实证表明，包括长时程任务及噪声/不一致标签场景，TraCeS在约束满足率和反馈效率上均优于基线。
原文摘要 · Abstract (English)
Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。