为视觉语言模型设计基于分类体系的评估方法，更准确衡量识别精度
Taxonomy-Aware Evaluation of Vision-Language Models
- 引入层级精确率与召回率，评估模型预测在分类体系中的正确性与具体程度
- 实验表明现有文本相似度指标无法有效捕捉分类体系中的语义关系
- 适用于细粒度图像分类任务，尤其适合评估模型对物种、类别等层级关系的理解
当视觉语言模型被要求识别图像中的实体时，可能回答‘我看到一棵针叶树’而非具体标签‘挪威云杉’。这带来两个评估挑战：一是将自由生成的文本映射到评价标签空间（如‘针叶树’）；二是分类指标应给予不精确但正确的回答部分得分（如‘挪威云杉’是‘针叶树’的一种）。为此，我们提出一种针对分类体系的评估框架，使用层级精确率与召回率来衡量模型预测在分类体系中的准确性与具体程度。实验表明，现有文本相似度方法难以捕捉分类体系中的语义相似性。我们进一步比较了多种将模型输出映射到分类体系的方法，并据此计算生成文本与真实标签之间的层级相似性。最后，基于该评估方案，我们分析了现代视觉语言模型在细粒度图像分类任务上的表现。
原文摘要 · Abstract (English)
When a vision-language model (VLM) is prompted to identify an entity depicted in an image, it may answer 'I see a conifer,' rather than the specific label 'norway spruce'. This raises two issues for evaluation: First, the unconstrained generated text needs to be mapped to the evaluation label space (i.e., 'conifer'). Second, a useful classification measure should give partial credit to less-specific, but not incorrect, answers ('norway spruce' being a type of 'conifer'). To meet these requirements, we propose a framework for evaluating unconstrained text predictions, such as those generated from a vision-language model, against a taxonomy. Specifically, we propose the use of hierarchical precision and recall measures to assess the level of correctness and specificity of predictions with regard to a taxonomy. Experimentally, we first show that existing text similarity measures do not capture taxonomic similarity well. We then develop and compare different methods to map textual VLM predictions onto a taxonomy. This allows us to compute hierarchical similarity measures between the generated text and the ground truth labels. Finally, we analyze modern VLMs on fine-grained visual classification tasks based on our proposed taxonomic evaluation scheme.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。