arXiv:2510.16185cs.LGcs.AI2025-10

用运行时监控语言提升奖励函数表达能力,让AI更懂复杂任务意图。

Expressive Reward Synthesis with the Runtime Monitoring Language

  • 基于运行时监控语言构建带记忆的奖励机器,支持非正则任务
  • 可实现计数、参数化条件等复杂奖励逻辑,突破传统限制
  • 适合需要精准行为控制的高风险场景,如自动驾驶

强化学习中的奖励误设问题常导致意外甚至有害行为。传统奖励函数将状态-动作对映射为标量值,缺乏解释性。奖励机器通过有限状态自动机表示奖励函数,实现结构化、非马尔可夫奖励。但其表达能力受限于正则语言,无法捕捉计数或参数化条件等复杂行为。本文基于运行时监控语言(RML)构建新型语言驱动的奖励机器,利用其内置记忆机制,可定义非正则、非马尔可夫任务的奖励函数。实验验证了该方法的表达能力,并在事件处理灵活性与任务建模方面优于现有方法。

原文摘要 · Abstract (English)

A key challenge in reinforcement learning (RL) is reward (mis)specification, whereby imprecisely defined reward functions can result in unintended, possibly harmful, behaviours. Indeed, reward functions in RL are typically treated as black-box mappings from state-action pairs to scalar values. While effective in many settings, this approach provides no information about why rewards are given, which can hinder learning and interpretability. Reward Machines address this issue by representing reward functions as finite state automata, enabling the specification of structured, non-Markovian reward functions. However, their expressivity is typically bounded by regular languages, leaving them unable to capture more complex behaviours such as counting or parametrised conditions. In this work, we build on the Runtime Monitoring Language (RML) to develop a novel class of language-based Reward Machines. By leveraging the built-in memory of RML, our approach can specify reward functions for non-regular, non-Markovian tasks. We demonstrate the expressiveness of our approach through experiments, highlighting additional advantages in flexible event-handling and task specification over existing Reward Machine-based methods.

强化学习奖励设计形式化验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。