构建3D几何视觉编码基准,评估AI写代码能力
Benchmarking PhD-Level Coding in 3D Geometric Computer Vision
- 设计博士级3D视觉编码任务,聚焦几何变换与算法实现
- 顶尖模型仅达36.6%通过率,显示当前能力与科研需求差距大
- 适合研究生成式AI在复杂科学计算中的应用与局限
AI辅助编程已深刻改变软件实践与研究流程,但现有模型在编写复杂3D几何视觉代码方面仍表现不佳。为衡量进展,我们提出GeoCodeBench——一个博士级基准,评估3D视觉编程能力。每个任务源自近年顶会论文的函数填空题:先由工具从官方仓库提取候选函数,再经人工筛选出核心3D几何组件。针对每项任务生成多样化的边界测试用例,支持全自动、可复现评分。我们评估了八种代表性开源与闭源模型。最优模型GPT-5仅达36.6%通过率,揭示当前能力与可靠3D科研编程间存在显著差距。GeoCodeBench将任务分为两层:通用3D能力(几何变换与光学/力学建模)与研究能力(新算法实现与几何逻辑路由)。得分在两轴正相关,但研究类任务更难。上下文消融实验表明,‘更多论文文本’并非总是更好:截断至方法部分的输入统计上优于全文输入,凸显长文本科学理解的未解难题。这些发现使GeoCodeBench成为推动通用编程向可信3D几何视觉编程演进的严格测试平台。
原文摘要 · Abstract (English)
AI-assisted coding has rapidly reshaped software practice and research workflows, yet today's models still struggle to produce correct code for complex 3D geometric vision. If models could reliably write such code, the research of our community would change substantially. To measure progress toward that goal, we introduce GeoCodeBench, a PhD-level benchmark that evaluates coding for 3D vision. Each problem is a fill-in-the-function implementation task curated from representative papers at recent venues: we first let a tool propose candidate functions from official repositories, then perform careful human screening to select core 3D geometric components. For every target, we generate diverse, edge-case unit tests, enabling fully automatic, reproducible scoring. We evaluate eight representative open- and closed-source models to reflect the current ecosystem. The best model, GPT-5, attains only 36.6% pass rate, revealing a large gap between current capabilities and dependable 3D scientific coding. GeoCodeBench organizes tasks into a two-level hierarchy: General 3D capability (geometric transformations and mechanics/optics formulation) and Research capability (novel algorithm implementation and geometric logic routing). Scores are positively correlated across these axes, but research-oriented tasks are markedly harder. Context ablations further show that "more paper text" is not always better: cutting off at the Method section statistically outperforms full-paper inputs, highlighting unresolved challenges in long-context scientific comprehension. Together, these findings position GeoCodeBench as a rigorous testbed for advancing from generic coding to trustworthy 3D geometric vision coding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。