用结构化世界状态预测奖励，跨领域泛化能力强。
Reward Prediction with Factorized World States
- 将观察转化为分层物体-属性结构，通过语义相似度估算奖励
- 在5个领域2454条轨迹上实现零样本奖励预测，误差比基线低60%
- 适合需要强泛化能力的智能体规划任务
智能体需推断动作后果并选择最大化奖励信号的动作。监督学习的奖励模型可能引入训练数据固有的偏差，限制对新目标和环境的泛化能力。本文探究仅依赖明确的世界状态表示是否可实现跨领域的准确奖励预测。为此，提出StateFactory方法，利用语言模型将非结构化观测转换为分层的物体-属性结构，使奖励可自然地以当前状态与目标状态间的语义相似度进行估计，并受层级约束。该紧凑的结构化表示显著提升了奖励泛化能力。在涵盖五个不同领域的新型基准数据集RewardPrediction上评估，该方法在零样本条件下相比VLWM-critic和LLM-as-a-Judge模型分别降低60%和8%的EPIC距离。更优的奖励质量也直接转化为更强的智能体规划性能，在AlfWorld上成功率达+21.64%，ScienceWorld上达+12.40%，优于反应式系统-1策略和系统-2规划代理。
原文摘要 · Abstract (English)
Agents must infer action outcomes and select actions that maximize a reward signal indicating how close the goal is to being reached. Supervised learning of reward models could introduce biases inherent to training data, limiting generalization to novel goals and environments. In this paper, we investigate whether well-defined world state representations alone can enable accurate reward prediction across domains. To address this, we introduce StateFactory, a factorized representation method that transforms unstructured observations into a hierarchical object-attribute structure using language models. This structured representation allows rewards to be estimated naturally as the semantic similarity between the current state and the goal state under hierarchical constraint. Overall, the compact representation structure induced by StateFactory enables strong reward generalization capabilities. We evaluate on RewardPrediction, a new benchmark dataset spanning five diverse domains and comprising 2,454 unique action-observation trajectories with step-wise ground-truth rewards. Our method shows promising zero-shot results against both VLWM-critic and LLM-as-a-Judge reward models, achieving 60% and 8% lower EPIC distance, respectively. Furthermore, this superior reward quality successfully translates into improved agent planning performance, yielding success rate gains of +21.64% on AlfWorld and +12.40% on ScienceWorld over reactive system-1 policies and enhancing system-2 agent planning. Project Page: https://statefactory.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。