构建视觉语言模型序数理解诊断基准，测试其对复杂路径大序号的推理能力。
OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
- 设计三轴控制难度的第N个物体识别任务，涵盖序数大小、路径复杂度和物体数量
- 包含39,000组问答对，支持零样本测试与步骤级轨迹一致性评估
- 揭示主流模型在大序号和复杂路径下严重退化，适合研究序列推理的开发者
视觉语言模型（VLMs）在多模态基准上已取得进展，但在序数理解方面仍存在明显短板，即难以追踪相对位置并泛化至大序号。我们提出OrdinalBench，一个标准化的诊断性基准，将序数理解作为评估任务。核心任务为第N个物体识别，由起始参考点和遍历规则定义。任务难度沿三个维度控制：(i) 序数量级，从较小数字到最大达300的极端情况；(ii) 排列复杂度，从单环路径到迷宫式路径；(iii) 物体总数。该基准提供39,000个问题-答案对，每对均标注真实推理轨迹，并在不同难度级别间平衡，支持可控的大规模测试。除答案评估外，框架要求模型生成结构化的逐步计数过程，配套开源评估工具可同时衡量最终准确率与步骤级路径一致性。对GPT-5、Gemini 2.5 Flash Lite、Qwen2.5-VL、InternVL3.5和Molmo的零样本测试显示，在大序号和复杂路径条件下性能急剧下降，凸显尽管在标准多模态任务中表现良好，但泛化能力依然薄弱。通过将序数理解设定为核心目标，OrdinalBench为开发具备更强序列推理能力的VLMs提供了可复现的基准与诊断框架。所有数据与代码可在https://ordinalbench.github.io/获取。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。