用大模型+视觉问答实现零样本细粒度物体识别，还能解释推理过程。
Help Me Identify: Is an LLM+VQA System All We Need to Identify Visual Concepts?
- 通过大模型生成物体属性描述，转为二元问题供视觉问答系统回答。
- 零样本下性能媲美现有方法，且推理过程完全可解释。
- 适合需要透明决策的场景，如医疗、自动驾驶等高可靠性应用。
人类能够从少量视觉数据中学习新物体，并在新情境下合理解释其存在或缺失的属性组合。受此启发，本文提出一种零样本细粒度视觉概念学习框架，结合大型语言模型（LLM）与视觉问答（VQA）系统。具体地，我们使用GPT-3对数据集中的视觉对象生成丰富的语言描述，将其转化为一组二元问题，再将这些问题与查询图像一同输入VQA系统，通过聚合答案判断测试图像中目标物体的存在与否。实验表明，该方法在零样本视觉分类和少样本概念学习任务上表现接近现有先进方法，且计算开销小，同时具备完整的可解释性。
原文摘要 · Abstract (English)
An ability to learn about new objects from a small amount of visual data and produce convincing linguistic justification about the presence/absence of certain concepts (that collectively compose the object) in novel scenarios is an important characteristic of human cognition. This is possible due to abstraction of attributes/properties that an object is composed of e.g. an object `bird' can be identified by the presence of a beak, feathers, legs, wings, etc. Inspired by this aspect of human reasoning, in this work, we present a zero-shot framework for fine-grained visual concept learning by leveraging large language model and Visual Question Answering (VQA) system. Specifically, we prompt GPT-3 to obtain a rich linguistic description of visual objects in the dataset. We convert the obtained concept descriptions into a set of binary questions. We pose these questions along with the query image to a VQA system and aggregate the answers to determine the presence or absence of an object in the test images. Our experiments demonstrate comparable performance with existing zero-shot visual classification methods and few-shot concept learning approaches, without substantial computational overhead, yet being fully explainable from the reasoning perspective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。