评测LLaMA2在多语言代码生成中的表现，发现简单任务可用，复杂计算需人工修正。
LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages
- 用典型科学计算问题测试代码生成、文档和测试能力
- 简单数值任务生成代码正确率高，复杂并行计算错误率显著上升
- 适合科研人员评估AI辅助编程效果，提示需警惕复杂场景
大型语言模型（LLMs）的快速发展为软件开发自动化带来了新可能。本文评估了Llama 2-70B模型在常用编程语言中自动化科学计算任务的能力。通过代表性测试问题，评估其生成代码、文档和单元测试的能力，以及在常见编程语言间翻译代码的能力。全面分析涵盖生成与翻译代码的编译结果、运行时行为及正确性。同时评估自动生成代码、文档和单元测试的质量。结果显示，尽管Llama 2-70B在简单数值任务上常生成语法正确且功能正常的代码，但在更复杂的并行或分布式计算中面临严重挑战，需大量人工修正。我们识别出关键局限，并提出未来改进方向，以更好利用AI自动化提升科学计算工作流效率。
原文摘要 · Abstract (English)
The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the Llama 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation and unit tests. Our results indicate that while Llama 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。