测试GPT-3.5在不同难度编程题上的表现，发现越难题正确率越低。
Evaluating ChatGPT-3.5 Efficiency in Solving Coding Problems of Different Complexity Levels: An Empirical Analysis
- 用提示工程提升模型解题能力，对简单题效果更明显。
- 易题正确率92%，中等题79%，难题仅51%。
- 擅长Python/Java/C++，对小众语言基本无能为力。
本文评估了ChatGPT的GPT-3.5-turbo模型在LeetCode平台上的编码问题解决能力，涵盖易、中、难三个难度等级。实验结果表明，该模型在易题上正确率为92%，中等题为79%，难题仅为51%。通过提示工程改进，链式思维提示带来14%-29%提升，提供失败测试用例反馈可提高38%-60%，切换至GPT-4则提升33%-58%。在语言支持方面，模型在Python中成功解决78%的问题，在Java为50%，在C++为50%，但在Elixir、Erlang和Racket中均未成功。研究验证了所有三个假设：难度越高解题率越低，提示工程在简单题上增益更大，且对主流编程语言表现优于小众语言。
原文摘要 · Abstract (English)
ChatGPT and other large language models (LLMs) promise to revolutionize software development by automatically generating code from program specifications. We assess the performance of ChatGPT's GPT-3.5-turbo model on LeetCode, a popular platform with algorithmic coding challenges for technical interview practice, across three difficulty levels: easy, medium, and hard. We test three main hypotheses. First, ChatGPT solves fewer problems as difficulty rises (Hypothesis 1). Second, prompt engineering improves ChatGPT's performance, with greater gains on easier problems and diminishing returns on harder ones (Hypothesis 2). Third, ChatGPT performs better in popular languages like Python, Java, and C++ than in less common ones like Elixir, Erlang, and Racket (Hypothesis 3). To investigate these hypotheses, we conduct automated experiments using Python scripts to generate prompts that instruct ChatGPT to create Python solutions. These solutions are stored and manually submitted on LeetCode to check their correctness. For Hypothesis 1, results show the GPT-3.5-turbo model successfully solves 92% of easy, 79% of medium, and 51% of hard problems. For Hypothesis 2, prompt engineering yields improvements: 14-29% for Chain of Thought Prompting, 38-60% by providing failed test cases in a second feedback prompt, and 33-58% by switching to GPT-4. From a random subset of problems ChatGPT solved in Python, it also solved 78% in Java, 50% in C++, and none in Elixir, Erlang, or Racket. These findings generally validate all three hypotheses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。