LIME精简评测集，提升多模态大模型评估效率与区分度。
LIME: Less Is More for MLLM Evaluation
- 通过半自动流程筛选图像理解任务，剔除无效样本和答案泄露
- 样本量减少76%，评估时间缩短77%，仍能有效区分模型性能
- 发现传统指标如CIDEr不适用于模型评分，应排除生成任务得分
多模态大语言模型（MLLMs）常在图像描述、视觉问答和推理等基准上进行评估。然而，许多基准包含过于简单或无信息量的样本，导致难以有效区分不同模型的表现。此外，跨多个基准评估带来巨大计算开销。为此，我们提出LIME（Less Is More for MLLM Evaluation），一个通过半自动化流程优化的高效评测基准。该流程剔除无信息样本并消除答案泄露，聚焦需要基于图像理解的任务。实验表明，LIME使样本数量减少76%，评估时间降低77%，同时更有效地揭示模型能力差异。值得注意的是，传统自动评价指标如CIDEr对模型描述任务表现评估不足；剔除描述任务得分后，整体性能评估更具准确性。所有代码与数据可在https://github.com/kangreen0210/LIME获取。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。