为大模型个性化能力设计多维度真实场景评测基准
Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models
- 构建涵盖6类生活场景的500+图像数据集，分年龄设计问答任务
- GPT-4o在年龄相关任务上准确率仅79%，表明模型仍有提升空间
- 首次按青年、中年、老年分层评估模型对不同人群需求的理解
大型多模态模型（LMMs）发展迅速，但现有评测无法全面客观评估其在真实场景中满足人类多样化需求的能力。为此，我们提出多维洞察（MDI）基准，包含超过500张覆盖六类日常生活的图像。该基准具有两大优势：一是每张图配简单与复杂两类问题，分别考察基础理解与深层推理；二是根据年龄分层设计问题，涵盖青年、中年、老年三类人群，以评估模型对不同群体需求的理解能力。实验显示，如GPT-4o等强模型在年龄相关任务上仅达79%准确率，表明现有模型在真实个性化应用方面仍有显著改进空间。我们预计该基准将推动大模型向更贴近真实世界个性化方向发展。MDI-Benchmark数据与代码已公开于https://mdi-benchmark.github.io/
原文摘要 · Abstract (English)
The rapidly developing field of large multimodal models (LMMs) has led to the emergence of diverse models with remarkable capabilities. However, existing benchmarks fail to comprehensively, objectively and accurately evaluate whether LMMs align with the diverse needs of humans in real-world scenarios. To bridge this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which includes over 500 images covering six common scenarios of human life. Notably, the MDI-Benchmark offers two significant advantages over existing evaluations: (1) Each image is accompanied by two types of questions: simple questions to assess the model's understanding of the image, and complex questions to evaluate the model's ability to analyze and reason beyond basic content. (2) Recognizing that people of different age groups have varying needs and perspectives when faced with the same scenario, our benchmark stratifies questions into three age categories: young people, middle-aged people, and older people. This design allows for a detailed assessment of LMMs' capabilities in meeting the preferences and needs of different age groups. With MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related tasks, indicating that existing LMMs still have considerable room for improvement in addressing real-world applications. Looking ahead, we anticipate that the MDI-Benchmark will open new pathways for aligning real-world personalization in LMMs. The MDI-Benchmark data and evaluation code are available at https://mdi-benchmark.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。