测试大模型数俯卧撑能力，发现其计数远不如识物，且易靠套路骗分。
PushupBench: Your VLM is not good at counting pushups
- 用446段长视频构建新评测基准，专门测重复动作计数。
- 顶尖模型准确率仅42.1%，开源小模型约6%，接近基线水平。
- 计数能力可提升模型时间推理，适合想练视频理解的开发者。
大型视觉语言模型能识别视频中‘发生了什么’，却难以准确统计‘发生了多少次’。我们提出PushupBench，包含446个长视频片段（平均36.7秒），用于评估重复动作计数能力。最先进模型在该任务上仅达42.1%的精确准确率；开源的40亿参数模型得分约6%，与监督基线相当。我们发现仅看准确率会误导判断——较弱模型依赖模态众数而非进行时间推理。在1000个样本上微调计数任务后，模型在MVBench（+2.15）、PerceptionTest（+1.88）、TVBench（+4.54）等通用视频理解评测中均取得提升，表明计数可作为更广泛时间推理能力的代理指标。PushupBench已集成至lmms-eval（https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262），并托管于pushupbench.com/
原文摘要 · Abstract (English)
Large vision-language models (VLMs) can recognize \textit{what} happens in video but fail to count \textit{how many} times. We introduce \textbf{PushupBench}, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1\% exact accuracy; open-source 4B models score $\sim$6\%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in \texttt{lmms-eval} (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。