通过两阶段方法提升大模型在细粒度视觉识别中的自由回答能力。
You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extraction
- 先问开放问题，再用约束解码提取答案
- 在数百至上千选项中准确识别目标类别
- 适合需要高精度细粒度分类的用户场景
尽管多模态大语言模型（MLLMs）推动了零样本视觉分类的发展，但评估自回归模型的自由回答仍具挑战。现有工作多聚焦于纯语言任务或不超过五选一的多项选择题，而细粒度视觉分类（FGVC）需面对数百至数千个高度相关的选项，且难以扩展到基于检索的问题。本文提出nlg2choice，一种两阶段方法：首先以开放式问题引导MLLM生成响应，再通过仅文本的约束解码预测最可能的选项。在检索场景中，采用提前停止法计算约束响应对应选项的概率，显著提升吞吐量。实验在七个多细粒度视觉数据集上验证了该方法在分类与检索任务中的性能优势，且在多种自然语言实现方式下均表现稳健。
原文摘要 · Abstract (English)
Despite the renewed interest in zero-shot visual classification due to the rise of Multimodal Large Language Models (MLLMs), the problem of evaluating free-form responses of auto-regressive models remains a persistent challenge. Most existing works focus on language-only tasks or don't consider Multiple Choice Questions (MCQs) beyond 5-way options, both of which are critical capabilities to solve tasks in Fine-Grained Visual Classification (FGVC) where choice counts are in the hundreds to thousands and the choices are highly related. Furthermore, in this highly multi-way MCQ setting it is not clear how to extend LLM choice extraction to retrieval-based problems, where computing probabilities over the choice set is computationally costly. In this work we investigate nlg2choice, a simple two-stage method which first asks the MLLM an open-ended question for the task with minimal constraints, then uses text-only constrained decoding to predict the most likely choice. In retrieval settings, we compute the probability of the constrained response taking that choice with an early stopping method to significantly improve throughput. Our results show improvement over a suite of seven fine-grained visual datasets when evaluating in terms of classification and retrieval, and show that this performance holds over the various ways that users of LLMs can implement tasks in natural language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。