用递归程序生成测试多模态模型的数学抽象能力
FractalBench: Diagnosing Visual-Mathematical Reasoning Through Recursive Program Synthesis
- 通过图像生成递归代码，检验模型从视觉模式中提炼数学规则的能力
- 4%模型正确捕捉数学结构，几何变换成功率17-21%，分支递归不足2%
- 适合评估大模型在复杂抽象推理中的真实水平
数学推理需要从视觉模式中抽象出符号规则——从有限推知无限。我们通过FractalBench基准，检验多模态大模型是否具备此能力，该基准要求从图像中合成分形程序。分形具有理想特性：仅需少数收缩映射，通过简单递归规则生成复杂自相似图案，迫使模型将视觉感知与数学抽象相结合。我们评估了四种领先多模态大模型（GPT-4o、Claude 3.7 Sonnet、Gemini 2.5 Flash、Qwen 2.5-VL）在12种经典分形上的表现。模型需生成可执行的Python代码重现分形，实现客观评估。结果揭示显著断层：76%生成语法正确代码，但仅4%正确捕捉数学结构。表现系统性差异——模型可处理几何变换（科赫曲线：17-21%），但在分支递归（如分形树）上失败率极高（<2%），暴露数学抽象的根本缺陷。FractalBench为视觉-数学推理提供抗污染诊断工具，项目地址：https://github.com/NaiveNeuron/FractalBench
原文摘要 · Abstract (English)
Mathematical reasoning requires abstracting symbolic rules from visual patterns -- inferring the infinite from the finite. We investigate whether multimodal AI systems possess this capability through FractalBench, a benchmark evaluating fractal program synthesis from images. Fractals provide ideal test cases: Iterated Function Systems with only a few contraction maps generate complex self-similar patterns through simple recursive rules, requiring models to bridge visual perception with mathematical abstraction. We evaluate four leading MLLMs -- GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Flash, and Qwen 2.5-VL -- on 12 canonical fractals. Models must generate executable Python code reproducing the fractal, enabling objective evaluation. Results reveal a striking disconnect: 76% generate syntactically valid code but only 4% capture mathematical structure. Success varies systematically -- models handle geometric transformations (Koch curves: 17-21%) but fail at branching recursion (trees: <2%), revealing fundamental gaps in mathematical abstraction. FractalBench provides a contamination-resistant diagnostic for visual-mathematical reasoning and is available at https://github.com/NaiveNeuron/FractalBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。