多目标强化学习需部署后仍获取奖励信号，否则策略失效
Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment
- 用环境状态与累计奖励拼接构建增强状态
- 部署后若无奖励信号，策略性能会下降
- 适用于需持续反馈的实时决策系统
本研究指出多目标强化学习（MORL）与传统单目标强化学习的关键差异：当使用非线性效用函数时，最优策略必须依赖当前环境状态和此前累积奖励的度量。通常通过将环境观测与过去奖励的折现和拼接，形成增强状态。尽管该方法在文献中广泛使用，但其隐含要求——部署后仍需持续访问奖励信号（或代理）——此前未被关注。本文解释了这一要求的根源，并探讨其实际影响。
原文摘要 · Abstract (English)
This research note identifies a previously overlooked distinction between multi-objective reinforcement learning (MORL), and more conventional single-objective reinforcement learning (RL). It has previously been noted that the optimal policy for an MORL agent with a non-linear utility function is required to be conditioned on both the current environmental state and on some measure of the previously accrued reward. This is generally implemented by concatenating the observed state of the environment with the discounted sum of previous rewards to create an augmented state. While augmented states have been widely-used in the MORL literature, one implication of their use has not previously been reported -- namely that they require the agent to have continued access to the reward signal (or a proxy thereof) after deployment, even if no further learning is required. This note explains why this is the case, and considers the practical repercussions of this requirement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。