测试大模型能否像人类侦探一样发现图像中的细微作弊线索。
CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning
- 设计新任务CaughtCheating，检验模型从图像中识别隐蔽作弊迹象的能力。
- GPT-o3在该任务上表现几乎为零，暴露出视觉推理短板。
- 适合研究多模态模型感知与推理边界的研究者参考。
近期的多模态大语言模型（MLLMs）如GPT-o3在现有基准上已接近满分表现，促使对更具挑战性的测试任务的需求增加。这些模型在部分人类专家级任务（如GeoGuesser）中表现出色，展现出通过图像中的微小线索进行连贯情境推理并得出可靠答案的潜力，类似优秀侦探。但它们能否达到人类侦探的水平？我们研究了若干困难场景，发现一种常见情况导致GPT-o3性能几乎归零，称之为CaughtCheating。该任务灵感来源于社交媒体上请求他人从伴侣分享的照片中检测可疑线索。我们进行了大量实验与分析，揭示当前MLLMs在解决此类任务时能力不足的根本原因。CaughtCheating提供了一类具有重要价值和实际应用意义的高难度视觉感知与推理任务，成功应对此类任务将推动MLLMs向具备人类级侦探感知与推理能力迈进。
原文摘要 · Abstract (English)
Recent agentic Multi-Modal Large Language Models (MLLMs) such as GPT-o3 have achieved near-ceiling scores on various existing benchmarks, motivating a demand for more challenging test tasks. These MLLMs have been reported to excel in a few expert-level tasks for humans, e.g., GeoGuesser, reflecting their potential as a detective who can notice minuscule cues in an image and weave them into coherent, situational explanations, leading to a reliable answer. But can they match the performance of excellent human detectives? To answer this question, we investigate some hard scenarios where GPT-o3 can still handle, and find a common scenario where o3's performance drops to nearly zero, which we name CaughtCheating. It is inspired by the social media requests that ask others to detect suspicious clues from photos shared by the poster's partner. We conduct extensive experiments and analysis to understand why existing MLLMs lack sufficient capability to solve this kind of task. CaughtCheating provides a class of challenging visual perception and reasoning tasks with great value and practical usage. Success in these tasks paves the way for MLLMs to acquire human-level detective perception and reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。