测试大模型生成数学动画代码时的逻辑与语法错误
ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation
- 构建专门评估动画代码生成的基准，关注时间精度和API兼容性
- 发现模型常出现无效函数调用和视觉逻辑偏差，错误率超30%
- 适合研究视频生成、教育AI和代码模型的开发者使用
传统基准如HumanEval和MBPP能有效检验代码逻辑与语法，但在生成动态教学动画时失效。本文提出ManiBench，一个专门评估大模型生成Manim CE代码性能的基准，重点关注时间准确性与版本感知的API正确性。该基准涵盖150-200道题，分五个难度等级，覆盖微积分、线性代数、概率、拓扑与人工智能，基于对3Blue1Brown的ManimGL源码（53,000行，143个场景类）的分析。采用四层评估框架，测量可执行性、版本冲突错误率、对齐得分与覆盖率。提供开源工具链，支持多模型与提示策略自动化评估。代码、数据与基准套件已在GitHub和Hugging Face公开。
原文摘要 · Abstract (English)
Traditional benchmarks like HumanEval and MBPP test logic and syntax effectively, but fail when code must produce dynamic, pedagogical visuals. We introduce ManiBench, a specialized benchmark evaluating LLM performance in generating Manim CE code, where temporal fidelity and version-aware API correctness are critical. ManiBench targets two key failure modes: Syntactic Hallucinations (valid Python referencing non-existent or deprecated Manim APIs) and Visual-Logic Drift (generated visuals diverging from intended mathematical logic through timing errors or missing causal relationships). The benchmark comprises 150-200 problems across five difficulty levels spanning calculus, linear algebra, probability, topology, and AI, grounded in analysis of 3Blue1Brown's ManimGL source (53,000 lines, 143 scene classes). Evaluation uses a four-tier framework measuring Executability, Version-Conflict Error Rate, Alignment Score, and Coverage Score. An open-source framework automates evaluation across multiple models and prompting strategies. Code, data and benchmark suite are available at https://github.com/nabin2004/ManiBench. and the dataset is hosted on https://huggingface.co/datasets/nabin2004/ManiBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。