用自然语言直接指导安全强化学习，自动理解约束并降低违规率。
From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning
- 用文本同时作为约束和训练信号，替代人工设计惩罚函数
- 在测试中违规率低于传统方法，且支持零样本迁移
- 适合需要灵活适应新约束的机器人、自动驾驶等场景
安全强化学习要求智能体在完成任务的同时遵守特定约束。以自然语言形式表达约束具有良好的实际应用潜力，因其具备灵活迁移和易用性。以往基于自然语言约束的安全强化学习方法通常需为每条约束手动设计代价函数，这既依赖领域知识又缺乏灵活性。本文利用文本在该任务中的双重作用——既作为约束输入，也作为训练信号——提出轨迹级文本约束翻译器（TTCT），用于替代人工设计的代价函数。实验表明，TTCT能有效理解文本约束与轨迹，由其训练出的策略相比标准代价函数具有更低的违反率。额外研究进一步验证了TTCT具备零样本迁移能力，可适应约束变化环境。
原文摘要 · Abstract (English)
Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。