用苹果芯片上的自动搜索框架，优化科学计算内核性能并检测隐藏错误。
Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon
- 基于苹果硅芯片的10个科学计算任务，构建可自动搜索内核的评测基准。
- 发现模型在训练数据范围内提升10.7倍，但跨尺寸测试时速度可能暴跌至0.23倍。
- 引入未见数据评分机制，能捕捉训练中无法察觉的隐蔽性错误。
我们提出 Metal-Sci，一个包含10个任务的科学计算内核基准，覆盖六种优化场景（如有限差分、多体问题、分子动力学等）。每个任务配备CPU参考实现、基于屋顶线模型的评估函数及未见过的泛化规模。配合轻量级自动搜索工具链，实时编译候选内核，多尺度评分，并将编译与正确性诊断反馈给固定大模型驱动的(1+1)进化搜索。在M1 Pro上对Claude Opus 4.7、Gemini 3.1 Pro和GPT 5.5进行单模型扫描，分布内自加速达1.00×至10.7×。关键方法论发现：最终使用未见配置的评分函数Φₜ，可有效识别隐蔽错误——例如某模板在未知维度返回错误结果，或某FFT3D内核在256³测试中从2.95×降至0.23×，此类问题仅靠分布内指标无法察觉。代码开源于https://github.com/vicgalle/metal-sci-kernels。
原文摘要 · Abstract (English)
We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in $n$-body problems, multi-field Boltzmann, neighbor-list molecular dynamics, multi-kernel PDE, FFT). Each task ships a CPU reference, a roofline-anchored fitness function, and a held-out generalization size. We pair the benchmark with a lightweight harness for automatic kernel search that runtime-compiles each candidate, scores it against the roofline across multiple sizes, and feeds structured compile and per-size correctness diagnostics back to a frozen LLM driving a $(1{+}1)$ evolutionary loop. We report matched single-model sweeps of Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 on M1 Pro: in-distribution self-speedups span $1.00\times$ to $10.7\times$. Beyond raw speedup, our central methodological claim is structural: the held-out gate scoring function $Φ_\mathcal{T}$ (evaluated once at end-of-run on a configuration the agent never sees during search) functions as a cheap mechanical oversight primitive on this automatic search loop, catching e.g. an Opus template <uint D> HMC win that returns wrong samples at unseen dimensions, and a GPT FFT3D best that wins in-distribution at $2.95\times$ speedup but collapses to $0.23\times$ on a $256^3$ held-out cube, a silent regression that the in-distribution score alone cannot see. Code at https://github.com/vicgalle/metal-sci-kernels
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。