视频大模型其实不会跟人，只会靠性别线索猜答案。
Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video
- 用九种诊断条件测试三类开源视频LLM，发现模型不跟踪具体人物。
- 换问题中角色名字后，模型仅4%-31%改变答案，对同性别角色无法区分。
- 模型依赖性别粗略线索，开放问答时准确率暴跌，适合用于评估模型真实能力。
视频大语言模型能否在长视频中持续追踪特定人物，准确报告其服装变化？当前基准测试显示，最强的7-8B开源模型在InfiniBench全局外观任务上得分达37-38%。但该成绩是否来自真正的角色追踪？我们通过九条件诊断协议测试三类架构不同的开源模型，并以Gemini 2.5 Flash为前沿参照。结果显示，当将问题中的角色名称更换为另一名演员时，模型仅4-31%改变答案，说明它们大多忽略提问对象。按性别拆解发现，换为异性别角色时反应更明显（差距13-28分），表明模型仅能捕捉粗粒度性别线索，无法区分同性别个体。进一步开放问答测试中，开源模型准确率下降18-25分，151个回答无一完全正确，而Gemini仅降12分。排除字幕、关键帧或帧数增加等干扰因素后，仍无法提升追踪表现，证明瓶颈在于模型如何将视觉信息与命名人物关联。研究发布诊断工具包，用于审计基准分数的真实含义。
原文摘要 · Abstract (English)
Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini~2.5~Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。