用对比学习提升智能体对动作可行性的判断能力。
CWM: Contrastive World Models for Action Feasibility Learning in Embodied Agent Pipelines
- 通过硬负样本对比训练,让模型区分物理上正确与看似合理但错误的动作。
- 在605组难题测试中,准确率提升6.76个百分点,AUC达到0.929。
- 适合需要高安全性的智能体任务,如机器人操作和复杂环境导航。
可靠的行动可行性评分器是具身智能体流水线中的关键瓶颈:在任何规划或推理之前,智能体必须识别哪些候选动作在当前状态下是物理可执行的。现有方法使用监督微调(SFT)训练动作评分器,但SFT将每个候选动作独立处理,未显式教导模型区分物理正确与微妙错误的动作。我们提出对比世界模型(CWM),利用InfoNCE对比目标和难挖掘负样本,对大型语言模型(LLM)进行动作评分器的微调。核心思想是在评分空间中将有效动作与无效动作分离,特别关注语义相似但物理不相容的难负样本。我们在ScienceWorld基准上进行了两项评估。首先,在605个难负样本测试对上的内在可操作性评估显示,CWM在最小编辑负样本的Precision@1上比SFT高出6.76个百分点,且AUC-ROC达到0.929(SFT为0.906)。其次,实时过滤特征研究衡量了任务执行过程中CWM对最优路径动作的排序表现,结果表明在分布外压力条件下,CWM的安全裕度(-2.39)显著优于SFT(-3.96),说明最优动作排名更靠前。这些结果支持假设:对比训练能生成更忠实反映物理可行性的表征。
原文摘要 · Abstract (English)
A reliable action feasibility scorer is a critical bottleneck in embodied agent pipelines: before any planning or reasoning occurs, the agent must identify which candidate actions are physically executable in the current state. Existing approaches use supervised fine-tuning (SFT) to train action scorers, but SFT treats each candidate independently and does not explicitly teach the model to discriminate between actions that are physically correct and those that are subtly wrong. We propose the Contrastive World Model (CWM), which fine-tunes a large language model (LLM) as an action scorer using an InfoNCE contrastive objective with hard-mined negative examples. The key idea is to push valid actions away from invalid ones in scoring space, with special emphasis on hard negatives: semantically similar but physically incompatible candidates. We evaluate CWM on the ScienceWorld benchmark through two studies. First, an intrinsic affordance evaluation on 605 hard-negative test pairs shows that CWM outperforms SFT by +6.76 percentage points on Precision@1 for minimal-edit negatives -- cases where a single word changes the physical outcome -- and achieves a higher AUC-ROC (0.929 vs. 0.906). Second, a live filter characterisation study measures how well CWM ranks gold-path actions against all valid environment actions during task execution. Under out-of-distribution stress conditions, CWM maintains a significantly better safety margin (-2.39) than SFT (-3.96), indicating that the gold action is ranked closer to the top. These results support the hypothesis that contrastive training induces representations that capture physical feasibility more faithfully than SFT alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。