对比4种大模型在4种推理架构下的视觉抽象推理表现,发现模型能力差异大于架构影响。
An Analysis of Architectural Impact on LLM-based Abstract Visual Reasoning: A Systematic Benchmark on RAVEN-FAIR
- 用四种推理架构测试主流大模型在RAVEN-FAIR上的表现
- GPT-4.1-Mini在所有架构中准确率最高,但架构效果因模型而异
- 多轮运行取最优值,避免单次评估结果不可靠
本研究系统评估了大语言模型(LLMs)在抽象视觉推理任务中的表现。选用GPT-4.1-Mini、Claude-3.5-Haiku、Gemini-1.5-Flash、Llama-3.3-70b四款模型,结合单次推理、嵌入控制重复、自省反思和多智能体四种推理架构,在RAVEN-FAIR数据集上进行测试。通过三阶段流程(JSON提取、LLM推理、工具函数)生成视觉响应,采用SSIM与LPIPS指标评估,分析思维链得分及错误类型(语义幻觉、数值误判)。结果显示,GPT-4.1-Mini在所有架构中均取得最高准确率,体现强推理能力;多智能体架构虽改变不同模型的语义与数值平衡,但效果不一致。各模型对架构设计敏感度各异,说明推理有效性具有显著模型特异性。响应覆盖范围差异成为干扰因素,使跨架构比较复杂。为估计每种配置上限,报告五次独立运行的最佳结果,符合近期建议——单次评估易受随机波动影响,结论不可靠。
原文摘要 · Abstract (English)
This study aims to systematically evaluate the performance of large language models (LLMs) in abstract visual reasoning problems. We examined four LLM models (GPT-4.1-Mini, Claude-3.5-Haiku, Gemini-1.5-Flash, Llama-3.3-70b) utilizing four different reasoning architectures (single-shot, embedding-controlled repetition, self-reflection, and multi-agent) on the RAVEN-FAIR dataset. Visual responses generated through a three-stage process (JSON extraction, LLM reasoning, and Tool Function) were evaluated using SSIM and LPIPS metrics; Chain-of-Thought scores and error types (semantic hallucination, numeric misperception) were analyzed. Results demonstrate that GPT-4.1-Mini consistently achieved the highest overall accuracy across all architectures, indicating a strong reasoning capability. While the multi-agent architecture occasionally altered semantic and numeric balance across models, these effects were not uniformly beneficial. Instead, each model exhibited distinct sensitivity patterns to architectural design, underscoring that reasoning effectiveness remains model-specific. Variations in response coverage further emerged as a confounding factor that complicates direct cross-architecture comparison. To estimate the upper-bound performance of each configuration, we report the best of five independent runs, representing a best-case scenario rather than an averaged outcome. This multi-run strategy aligns with recent recommendations, which emphasize that single-run evaluations are fragile and may lead to unreliable conclusions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。