对比五款大模型生成代码能力，ChatGPT表现最佳。
Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation
- 用LeetCode题目测试模型，评估生成代码质量。
- ChatGPT正确率远超Code Llama等专精模型。
- 长提示下错误增多，适合关注代码生成的开发者。
近年来，大型语言模型（LLMs）在软件工程等领域展现出巨大潜力。本研究评估了五种前沿LLM（Bard、BingChat、ChatGPT、Llama2、Code Llama）在文本转代码任务中的表现。通过LeetCode网站获取编程问题描述，以Python形式生成解决方案，并利用LeetCode的测试功能评估输出质量。结果显示，各模型性能差异显著，ChatGPT在处理典型编程挑战时表现最优，甚至超越专为代码优化的Code Llama。进一步分析生成代码的运行时间与内存使用，发现其优于多数提交。详细错误分析显示，缩进和格式错误是主要问题，且随着输入提示长度增加，错误率明显上升。结果揭示了模型在复杂上下文下的局限性。
原文摘要 · Abstract (English)
In recent years, large language models (LLMs) have emerged as powerful tools with potential applications in various fields, including software engineering. Within the scope of this research, we evaluate five different state-of-the-art LLMs - Bard, BingChat, ChatGPT, Llama2, and Code Llama - concerning their capabilities for text-to-code generation. In an empirical study, we feed prompts with textual descriptions of coding problems sourced from the programming website LeetCode to the models with the task of creating solutions in Python. Subsequently, the quality of the generated outputs is assessed using the testing functionalities of LeetCode. The results indicate large differences in performance between the investigated models. ChatGPT can handle these typical programming challenges by far the most effectively, surpassing even code-specialized models like Code Llama. To gain further insights, we measure the runtime as well as the memory usage of the generated outputs and compared them to the other code submissions on Leetcode. A detailed error analysis, encompassing a comparison of the differences concerning correct indentation and form of the generated code as well as an assignment of the incorrectly solved tasks to certain error categories allows us to obtain a more nuanced picture of the results and potential for improvement. The results also show a clear pattern of increasingly incorrect produced code when the models are facing a lot of context in the form of longer prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。