arXiv:2512.23739cs.CLcs.AI2025-12

让机器人学会猜餐具藏在哪,靠视觉+语言模型推理。

Break Out the Silverware -- Semantic Understanding of Stored Household Items

  • 用视觉理解+大模型推理,把物品位置转成自然语言描述
  • 在真实厨房数据上准确率接近人类水平
  • 适合研究家庭机器人认知能力的学者和开发者

机器人听懂‘拿个盘子’却不知它藏在哪——这是家用服务机器人的常见难题。本文提出‘存储物品挑战’基准任务,评估机器人推断物品隐藏位置的能力。包含两个数据集:100组真实厨房图像与人工标注的存放位置(真实世界评估集),以及6500组带存储区域多边形标注的公开厨房图像(开发集)。为应对挑战,提出NOAM模型:先将视觉输入转化为空间上下文描述,再通过大语言模型(如GPT-4)推断最可能的隐藏位置。该混合式代理在多项基线(随机选择、视觉语言模型、Gemini等)中显著提升准确率,逼近人类表现,适用于模块化部署于复杂家庭机器人系统。

原文摘要 · Abstract (English)

``Bring me a plate.'' For domestic service robots, this simple command reveals a complex challenge: inferring where everyday items are stored, often out of sight in drawers, cabinets, or closets. Despite advances in vision and manipulation, robots still lack the commonsense reasoning needed to complete this task. We introduce the Stored Household Item Challenge, a benchmark task for evaluating service robots' cognitive capabilities: given a household scene and a queried item, predict its most likely storage location. Our benchmark includes two datasets: (1) a real-world evaluation set of 100 item-image pairs with human-annotated ground truth from participants' kitchens, and (2) a development set of 6,500 item-image pairs annotated with storage polygons over public kitchen images. These datasets support realistic modeling of household organization and enable comparative evaluation across agent architectures. To begin tackling this challenge, we introduce NOAM (Non-visible Object Allocation Model), a hybrid agent pipeline that combines structured scene understanding with large language model inference. NOAM converts visual input into natural language descriptions of spatial context and visible containers, then prompts a language model (e.g., GPT-4) to infer the most likely hidden storage location. This integrated vision-language agent exhibits emergent commonsense reasoning and is designed for modular deployment within broader robotic systems. We evaluate NOAM against baselines including random selection, vision-language pipelines (Grounding-DINO + SAM), leading multimodal models (e.g., Gemini, GPT-4o, Kosmos-2, LLaMA, Qwen), and human performance. NOAM significantly improves prediction accuracy and approaches human-level results, highlighting best practices for deploying cognitively capable agents in domestic environments.

机器人认知视觉语言模型家居自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。