arXiv:2606.14240cs.AI2026-06

用20个问题游戏测试模型从物理属性推断物体功能的能力。

AFFORDANCE20Q: Evaluating Affordance Reasoning from Physical Properties

论文配图:AFFORDANCE20Q: Evaluating Affordance Reasoning from Physical Properties
图 1 · 摘自论文原文
  • 设计20问游戏,不暴露物体身份,强制模型基于物理属性推理。
  • 15个顶尖大模型表现远低于人类,差距约20分,且后期提问缺乏区分度。
  • 提出基于知识库的规则生成方法,可提升开源模型性能15.2分。

物体的可用性推理(即从形状、材质等物理属性推断其可能动作)是人类物理理解的核心,对大语言模型也日益重要。现有基准大多在评测中暴露物体身份,导致模型依赖记忆而非真实推理。为此,我们提出Affordance20Q,一种以20问游戏形式展开的新型可用性推理基准,不透露物体身份。模型需通过询问关于物理属性的非/是问题,从候选集中识别隐藏物体的可用性。该数据集包含454个物体、59种可用性、共1,009场游戏,均经人工筛选与标注。我们在15个前沿大模型上进行实验,发现其性能相比人类低约20分。基于KL的信息增益分析显示,模型在游戏后期难以提出有区分度的问题。为缩小差距,我们提出基于知识库的规则归纳方法KARI,利用知识库证据生成可用性规则。KARI使开源模型最高提升15.2分,但知识库覆盖不足限制进一步提升。代码与数据已开源。

原文摘要 · Abstract (English)

Affordance reasoning, the inference of an object's action possibilities from its physical properties (e.g., shape and material), is fundamental to human physical understanding and increasingly critical for Large Language Models (LLMs). However, existing affordance benchmarks largely expose explicit object identities in the evaluation setup, allowing models to rely on memorized object-affordance mappings rather than reasoning over physical properties. To address this gap, we introduce Affordance20Q, a novel affordance reasoning benchmark formulated as a 20-Questions game without exposing the object's identity. In each game, the model identifies a hidden object's affordance from a candidate set by asking yes/no questions about its physical properties. Affordance20Q comprises 1,009 games over 454 objects and 59 affordances, all manually filtered, refined, and annotated. We conduct comprehensive experiments with 15 state-of-the-art LLMs and find a substantial gap (~20 points) compared to human performance. A KL-based information-gain (IG) analysis further shows that models fail to ask discriminating questions as the game progresses. To close the gap, we develop KB-Anchored Rule Induction (KARI), a pipeline based on LLMs that generates affordance rules grounded in evidence from knowledge bases (KBs). KARI improves open-source LLMs by up to 15.2 points, while the limited coverage of KBs hinders further gains. We release all our code and data at https://github.com/1171-jpg/Affordance20Q.git.

可用性推理大模型评测20问游戏知识库

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。