提出方法识别并剔除规划中的虚假目标，提升智能体决策安全性和可靠性。
Rejecting Hallucinated State Targets during Planning
- 通过学习一个可行性评估器，判断生成目标是否可行
- 实验显示可显著减少幻觉行为并提升多种智能体性能
- 无需修改原智能体，适配性强，适合高安全性场景
在计算决策智能体的规划过程中，生成或预测模型常作为“生成器”提出代表预期或理想状态的“目标”。然而，学习模型不可避免会产生不可行的目标，导致幻觉行为和安全风险。本文首先分析生成器可能产生的不可行目标类型，随后设计一种策略，通过学习目标可行性评估器来识别并拒绝不可行目标。为确保评估器稳健且不产生幻觉，采用结合离策略兼容学习规则、分布式架构及基于事后重标注的数据增强的设计方案。该评估器可附加于规划智能体上，通过观察智能体与环境的交互及其生成器产出的目标进行学习，无需更改原智能体或其生成器。受控实验表明，该方法能显著降低幻觉行为，并在多种现有智能体中实现性能提升。
原文摘要 · Abstract (English)
In planning processes of computational decision-making agents, generative or predictive models are often used as "generators" to propose "targets" representing sets of expected or desirable states. Unfortunately, learned models inevitably hallucinate infeasible targets that can cause delusional behaviors and safety concerns. We first investigate the kinds of infeasible targets that generators can hallucinate. Then, we devise a strategy to identify and reject infeasible targets by learning a target feasibility evaluator. To ensure that the evaluator is robust and non-delusional, we adopted a design choice combining off-policy compatible learning rule, distributional architecture, and data augmentation based on hindsight relabeling. Attaching to a planning agent, the designed evaluator learns by observing the agent's interactions with the environment and the targets produced by its generator, without the need to change the agent or its generator. Our controlled experiments show significant reductions in delusional behaviors and performance improvements for various kinds of existing agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。