首个专用于向量指令生成的评测基准，揭示大模型在高性能编程中的短板。
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
- 构建136个任务的基准，覆盖五类主流向量指令集。
- 18个大模型在向量化代码生成上正确率普遍低于标量代码。
- 开源数据集助力高性能计算领域模型优化研究。
SIMD（单指令多数据）指令及其编译器内联函数被现代处理器广泛支持，用于加速性能敏感型任务。向量内联编程在开发效率与性能之间权衡，广泛应用于主流高性能库和日常计算任务中。大型语言模型（LLMs）在代码生成方面表现出强大能力，有望辅助程序员应对向量内联编程的挑战。然而，现有代码生成评测仅关注标量代码，尚不清楚大模型在生成使用SIMD内联的向量化代码时的表现。为填补这一空白，我们提出SimdBench，首个专门针对SIMD内联代码生成的基准，包含136个精心设计的任务，涵盖五种代表性SIMD内联：SSE（x86流式向量扩展）、AVX（x86高级向量扩展）、Neon（ARM高级向量扩展）、SVE（ARM可扩展向量扩展）和RVV（RISC-V向量扩展）。我们对18个代表性大模型在SimdBench上进行了系统性评估（衡量正确性和性能），得出一系列新颖且深刻的发现。评估结果表明，大模型在向量内联代码生成中的pass@k普遍低于标量代码生成。深入分析揭示了大模型在该挑战性领域进一步发展的潜在方向。SimdBench已开源，网址为https://anonymous.4open.science/r/SimdBench-1B3F/，以促进更广泛的研究社区发展。
原文摘要 · Abstract (English)
SIMD (Single Instruction Multiple Data) instructions and their compiler intrinsics are widely supported by modern processors to accelerate performance-critical tasks. SIMD intrinsic programming, a trade-off between coding productivity and high performance, is widely used in the development of mainstream performance-critical libraries and daily computing tasks. Large Language Models (LLMs), which have demonstrated strong and comprehensive capabilities in code generation, show promise in assisting programmers with the challenges of SIMD intrinsic programming. However, existing code-generation benchmarks focus on only scalar code, and it is unclear how LLMs perform in generating vectorized code using SIMD intrinsics. To fill this gap, we propose SimdBench, the first code benchmark specifically designed for SIMD-intrinsic code generation, comprising 136 carefully crafted tasks and targeting five representative SIMD intrinsics: SSE (x86 Streaming SIMD Extension), AVX (x86 Advanced Vector Extension), Neon (ARM Advanced SIMD Extension), SVE (ARM Scalable Vector Extension), and RVV (RISC-V Vector Extension). We conduct a systematic evaluation (measuring both correctness and performance) of 18 representative LLMs on SimdBench, resulting in a series of novel and insightful findings. Our evaluation results demonstrate that LLMs exhibit a universal decrease in pass@k during SIMD-intrinsic code generation compared to scalar-code generation. Our in-depth analysis highlights promising directions for the further advancement of LLMs in the challenging domain of SIMD-intrinsic code generation. SimdBench is fully open source at https://anonymous.4open.science/r/SimdBench-1B3F/ to benefit the broader research community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。