评测大模型跨并行编程接口翻译的可靠性,发现其存在严重语义失真问题。
ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation
- 以计算核为核心,通过可复现的规范定义测试环境
- 多接口间翻译失败率高,且对代码扰动敏感
- 适合研究大模型代码生成与高性能计算迁移的学者
现代计算密集型软件需在不断变化的加速器生态中迁移,涉及CUDA、OpenMP、OpenCL及OpenMP目标卸载等编程接口。大语言模型和自主编码代理被用于此类迁移,但缺乏可靠方法评估其是否保留线程索引、同步、内存管理、主机-设备协同等底层并行语义。本文提出ParBench,一个以计算核为中心的基准框架,通过声明式规范固定构建、运行与验证环境,仅要求模型翻译计算核。该框架基于多个开源高性能计算套件,覆盖CUDA、OpenMP、OpenCL及OpenMP目标卸载间的典型跨接口翻译方向。为区分真实翻译能力与表面记忆,引入基于抽象语法树(AST)的语义保持型源码增强。对主流开源与专有大模型的评估显示,存在方向不对称、多文件协调缺失、API适配不全、对源码扰动鲁棒性差等持续障碍。代码见https://github.com/Scientific-Computing-Lab/ParBench。
原文摘要 · Abstract (English)
Modern compute-intensive software must migrate across a changing ecosystem of accelerators, programming APIs, compiler stacks, and portability layers, including CUDA, OpenMP, OpenCL, and OpenMP target offload. Large language models and autonomous coding agents are increasingly proposed for such migration, but the field lacks reliable ways to measure whether they preserve the low-level parallel semantics that make translations behaviorally valid, including thread indexing, synchronization, memory management, host-device coordination, and API-specific execution structure. We present ParBench, a kernel-centric benchmark framework for evaluating LLM-based parallel API translation under executable, reproducible conditions. ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only the computational kernels. It draws on multiple open-source HPC suites and covers representative cross-API translation directions among CUDA, OpenMP, OpenCL, and OpenMP target offload. To test whether success reflects robust translation rather than surface-form memorization, ParBench includes AST-driven, intended behavior-preserving, baseline-validated source augmentation. Evaluations on state-of-the-art open and proprietary LLMs show persistent barriers to reliable parallel code translation, including direction asymmetry, multi-file coordination, incomplete API adaptation, and uneven robustness to source-level perturbations. Code is available at https://github.com/Scientific-Computing-Lab/ParBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。