用模块化设计模拟人类解谜能力，精准预测图像中的时间和地点。
PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction
- 分模块构建解谜流程：感知、推理、组合、检索、去噪
- 在TARA和WikiTilo上超越大模型32%以上，达最新水平
- 零样本可解释性强，适合需要透明决策的场景
从图像中预测时间与位置是一项复杂任务，需具备类人解谜能力。本文将此能力分解为核心技能，并通过名为PuzzleGPT的专家流水线实现。该系统包含视觉感知模块识别线索、推理模块生成候选结果、组合模块整合多线索信息、网络检索模块获取外部知识、噪声过滤模块提升鲁棒性。该方法为零样本、可解释且稳健，在TARA和WikiTilo两个数据集上达到当前最优性能，显著优于BLIP-2、InstructBLIP、LLaVA等大型视觉语言模型，以及VisProg等自动生成推理管道，提升幅度分别达32%和38%，甚至可媲美或超过微调模型。
原文摘要 · Abstract (English)
The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。