用儿童认知任务测试视觉语言模型,发现模型表现与孩子不完全一致
LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")

- 基于儿童认知数据构建多尺度评测基准
- 大模型整体准确率高但错误模式与孩子不匹配
- 小模型反而在部分任务上更像低龄儿童
鉴于人类经验的固有多模态特性,视觉语言模型(VLMs)在建模认知发展方面具有巨大潜力。实现这一潜力需要工具来跨任务、年龄和人群比较VLMs与人类认知发展。本文提出LEVANTE-bench,基于学习变异性网络(LEVANTE)的任务与数据,该网络公开了跨语言和文化的儿童认知测量数据。在LEVANTE-bench中,我们系统评估了多个VLMs在六个任务上的表现,将其与来自三个国家的5-12岁儿童(共1547名)进行对比。评估覆盖多个尺度:整体准确率、任务与题项级对齐程度,以及试次级错误分布匹配度。结果显示,不同尺度上的对齐性差异显著:在任务和题项层面,能力更强的模型更接近人类;但在错误分布匹配方面,不同任务间差异大,且部分任务中较小的模型反而更贴近年幼儿童的错误模式。此外,即使是最优模型在矩阵推理和心理旋转任务上仍表现不佳。因此,当前VLM架构仅部分反映儿童的认知能力。
原文摘要 · Abstract (English)
Given the inherently multimodal nature of human experience, vision-language models (VLMs) hold substantial promise for modeling human cognition as it grows and develops with experience. Realizing their potential requires tools for comparing VLMs with human cognitive development across tasks, ages, and populations. We present LEVANTE-bench, a benchmark based on tasks and data from the Learning Variability Network (LEVANTE), which distributes open-source tasks and data measuring children's cognition across languages and cultures. In LEVANTE-bench, we systematically assess VLMs on six tasks, comparing their alignment with children aged 5-12 ($N$ = 1547) across three countries. We compare models at multiple scales, assessing their overall accuracy, their task- and item-level alignment with children, and how well they match children's trial-level error distributions. Alignment was heterogeneous across scales: at the level of tasks and items, more capable models aligned better with humans. However, match to human error distributions varied widely across tasks, and for several tasks, smaller models matched younger children's errors better. In addition, even the best-performing VLMs struggled on matrix reasoning and mental rotation tasks. Thus, current VLM architectures align only partially with the cognitive abilities of children.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。