用少量数据高效评估大模型，准确率超96%。
Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models
- 用最远点采样法选取代表性数据子集
- 仅用1%数据保持96%以上相关性
- 适合需要快速评估大模型的团队
我们提出一种高效的大型视觉语言模型（VLM）评估协议。由于这些模型具备广泛的知识和推理能力，需多个基准进行全面评估，导致计算成本高昂。为提升效率，我们构建了一个子集，其结果与完整基准评估相当。基准分类实验表明，单一基准无法覆盖所有挑战。我们引入基于最远点采样（FPS）的子集构建方法。实验显示，基于FPS的基准与完整评估具有强相关性（>0.96），且仅使用约1%的数据。此外，将FPS应用于现有基准可提升与整体评估结果的相关性，表明其有潜力减少无意的数据集偏差。
原文摘要 · Abstract (English)
We propose an efficient evaluation protocol for large vision-language models (VLMs). Given their broad knowledge and reasoning capabilities, multiple benchmarks are needed for comprehensive assessment, making evaluation computationally expensive. To improve efficiency, we construct a subset that yields results comparable to full benchmark evaluations. Our benchmark classification experiments reveal that no single benchmark fully covers all challenges. We then introduce a subset construction method using farthest point sampling (FPS). Our experiments show that FPS-based benchmarks maintain a strong correlation (> 0.96) with full evaluations while using only ~1\% of the data. Additionally, applying FPS to an existing benchmark improves correlation with overall evaluation results, suggesting its potential to reduce unintended dataset biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。