评测大模型解力扣题表现，揭示其优劣与改进方向。
Performance Review on LLM for solving leetcode problems
- 用爬虫收集力扣题目，让GPT-4、GPT-3.5生成代码解题。
- 通过pass@k指标评估成功率，发现GPT-4在多轮尝试下正确率更高。
- 适合研究自动化编程助手的开发者和教育者参考。
本文对大型语言模型（LLMs）在解决力扣（Leetcode）编程挑战中的表现进行了全面评估。我们通过爬取力扣网站，收集涵盖不同难度和主题的多样化题目，并使用GPT-4与GPT-3.5-turbo（ChatGPT-turbo）生成代码解决方案。针对生成结果，系统性地评估了代码正确性与运行效率，采用pass@k指标衡量在有限尝试次数内的成功概率，并分析了解决方案的执行时间。实验结果揭示了当前大模型在代码生成与问题求解任务中的优势与局限，为自动化编程辅助工具的应用前景及改进方向提供了深入见解。
原文摘要 · Abstract (English)
This paper presents a comprehensive performance evaluation of Large Language Models (LLMs) in solving programming challenges from Leetcode, a widely used platform for algorithm practice and technical interviews. We began by crawling the Leetcode website to collect a diverse set of problems encompassing various difficulty levels and topics. Using this dataset, we generated solutions with multiple LLMs, including GPT-4 and GPT-3.5-turbo (ChatGPT-turbo). The generated solutions were systematically evaluated for correctness and efficiency. We employed the pass@k metric to assess the success rates within a given number of attempts and analyzed the runtime performance of the solutions. Our results highlight the strengths and limitations of current LLMs [10] in code generation and problem-solving tasks, providing insights into their potential applications and areas for improvement in automated programming assistance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。