提出评估多模态推理链质量的新框架，更贴近人类判断。
MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps
- 基于图像描述与推理步骤的细粒度标注数据集
- 四款先进多模态大模型测试，结果更接近人工判断
- 适合研究多模态推理可解释性与评估方法的学者
多模态思维链（MCoT）是一种提升多模态大语言模型（MLLMs）在复杂推理任务中表现的流行提示策略。然而，目前缺乏自动化方法来评估MCoT中推理步骤的质量。为此，我们提出多模态思维链评估框架（MiCEval），通过评估图像描述质量和每一步推理的正确性、相关性和信息量来衡量推理链整体质量。该框架基于一个细粒度标注数据集，其中每个推理步骤均按准确性、相关性和信息量进行评分。在四个最先进的MLLM上进行的大量实验表明，使用MiCEval进行逐步评估的结果比基于余弦相似度或微调的方法更接近人类判断。代码与数据集可在 https://github.com/alenai97/MiCEval 获取。
原文摘要 · Abstract (English)
Multimodal Chain of Thought (MCoT) is a popular prompting strategy for improving the performance of multimodal large language models (MLLMs) across a range of complex reasoning tasks. Despite its popularity, there is a notable absence of automated methods for evaluating the quality of reasoning steps in MCoT. To address this gap, we propose Multimodal Chain-of-Thought Evaluation (MiCEval), a framework designed to assess the correctness of reasoning chains by evaluating the quality of both the description and each reasoning step. The evaluation of the description component focuses on the accuracy of the image descriptions, while the reasoning step evaluates the quality of each step as it is conditionally generated based on the preceding steps. MiCEval is built upon a fine-grained dataset with annotations that rate each step according to correctness, relevance, and informativeness. Extensive experiments on four state-of-the-art MLLMs show that step-wise evaluations using MiCEval align more closely with human judgments compared to existing methods based on cosine similarity or fine-tuning approaches. MiCEval datasets and code can be found in https://github.com/alenai97/MiCEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。