arXiv:2505.22146cs.CVcs.AI2025-05

用低维属性对齐视觉与语言,实现类人工具选择

Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language

  • 通过13个属性构建跨模态对齐框架,连接工具图像与任务描述
  • 74%准确率超越传统方法,接近GPT-4o但参数更少
  • 抓握性等操作属性最关键,适合需要可解释性的应用

灵活工具选择体现人类独特认知能力,但现有计算模型仍不完善。本文构建包含115种常见工具的ToolNet数据集,每种工具标注13个涵盖物理、功能和心理属性的特征,并配以自然语言使用场景。利用ResNet或ViT提取图像属性,结合微调后的GPT-2、LLaMA、DeepSeek等语言模型从任务描述中推断所需属性。该方法在工具选择任务中达74%准确率,显著优于直接匹配(20%)及小型多模态模型(21%-58%),且接近大模型GPT-4o(73%)表现,但参数量大幅减少。人类评估验证其符合人类决策模式,泛化实验显示对新工具类别仍有良好表现。消融分析表明,抓握性、伸长性、手相关性等操作属性在跨模态中始终最关键。本工作提供了一种参数高效、可解释的类人工具认知方案,推动认知科学理解与实际应用。

原文摘要 · Abstract (English)

Flexible tool selection reflects a complex cognitive ability that distinguishes humans from other species, yet computational models that capture this ability remain underdeveloped. We developed a framework using low-dimensional attribute representations to bridge visual tool perception and linguistic task understanding. We constructed a comprehensive dataset (ToolNet) containing 115 common tools labeled with 13 carefully designed attributes spanning physical, functional, and psychological properties, paired with natural language scenarios describing tool usage. Visual encoders (ResNet or ViT) extract attributes from tool images while fine-tuned language models (GPT-2, LLaMA, DeepSeek) derive required attributes from task descriptions. Our approach achieves 74% accuracy in tool selection tasks-significantly outperforming direct tool matching (20%) and smaller multimodal models (21%-58%), while approaching performance of much larger models like GPT-4o (73%) with substantially fewer parameters. Human evaluation studies validate our framework's alignment with human decision-making patterns, and generalization experiments demonstrate effective performance on novel tool categories. Ablation studies revealed that manipulation-related attributes (graspability, elongation, hand-relatedness) consistently prove most critical across modalities. This work provides a parameter-efficient, interpretable solution that mimics human-like tool cognition, advancing both cognitive science understanding and practical applications in tool selection tasks.

工具选择多模态对齐可解释性属性建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。