用决策树分解视觉分类，测试大模型的结构化推理能力
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
- 将分类任务拆解为决策树路径，提升可解释性
- 在GTSRB数据集上仅达98.2%知识理解率，但性能仍低于标准提示
- 图像描述能有效提升树推理与零样本提示的表现
视觉语言模型（VLMs）在零样本视觉分类中表现优异，但在细粒度任务和大规模层次标签空间中的表现尚不明确。本文探究结构化树状推理是否能提升性能。提出一种框架，将分类分解为可解释的决策路径，基于决策树在细粒度（GTSRB）和粗粒度（CIFAR-10）数据集上进行评估。尽管模型对树知识的理解准确率达98.2%，但树状推理始终落后于标准零样本提示。进一步尝试通过大语言模型生成类别名和图像描述增强树提示，发现图像描述能有效提升树推理与零样本方法的性能。研究揭示了结构化推理在视觉分类中的局限性，并为构建更可解释的VLM系统提供参考。
原文摘要 · Abstract (English)
Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。