通过识别瓶颈状态推断用户隐含目标，减少查询次数实现对齐行为。
Inferring Implicit Goals Across Differing Task Models
- 基于马尔可夫决策过程，用瓶颈状态定位潜在隐含子目标。
- 在多个任务中成功推断未明说的目标，且仅需最少查询次数。
- 适合需要理解用户隐性需求的智能体系统设计者。
生成与用户价值观一致的行为面临一大挑战：不仅要考虑明确的用户目标，还需应对未明确表达的隐含需求。当用户对任务模型的理解与智能体估计不一致时，这种隐含需求尤为常见，可能导致用户错误预期某些行为是必然或保证发生的。本文针对此类期望错配问题，将未指定的用户子目标纳入马尔可夫决策过程（MDP）框架，并按需进行查询以识别。方法通过识别瓶颈状态作为潜在隐含子目标的候选，引入一种最小化查询次数的策略，确保找到能达成底层目标的策略。实验表明，该方法在多种任务中有效推断并实现未陈述的目标。
原文摘要 · Abstract (English)
One of the significant challenges to generating value-aligned behavior is to not only account for the specified user objectives but also any implicit or unspecified user requirements. The existence of such implicit requirements could be particularly common in settings where the user's understanding of the task model may differ from the agent's estimate of the model. Under this scenario, the user may incorrectly expect some agent behavior to be inevitable or guaranteed. This paper addresses such expectation mismatch in the presence of differing models by capturing the possibility of unspecified user subgoal in the context of a task captured as a Markov Decision Process (MDP) and querying for it as required. Our method identifies bottleneck states and uses them as candidates for potential implicit subgoals. We then introduce a querying strategy that will generate the minimal number of queries required to identify a policy guaranteed to achieve the underlying goal. Our empirical evaluations demonstrate the effectiveness of our approach in inferring and achieving unstated goals across various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。