提出可评估代码质量的LLM框架，帮开发者更好用大模型写高质量代码。
PELLI: Framework to effectively integrate LLMs for quality software generation
- 构建迭代式框架PELLI，量化评估维护性、性能、可靠性三方面代码质量。
- 对比五款主流LLM，GPT-4T与Gemini在三项指标上表现更优。
- 强调提示词设计影响代码质量，适配不同应用场景提升实际应用价值。
近期研究发现，当大语言模型（LLM）被恰当提示和配置时，其生成结果表现参差不齐，但常能达到或超过基线水平。然而现有对比存在两大缺陷：仅以可靠性为评估指标，且仅选取少数模型（如Codex和ChatGPT）。本文提出一个全面的代码质量评估框架——程序卓越通过LLM迭代（PELLI），该框架基于迭代分析过程，确保高质量代码变更。我们扩展了当前研究，对五款主流LLM进行综合评估，生成关于维护性、性能和可靠性三类非功能性需求的定量指标。实验覆盖三个应用领域，并遵循Python编码规范。结果显示，GPT-4T与Gemini在三项指标上表现略优；提示词设计显著影响代码质量；不同应用领域在各指标上得分差异明显，同一指标下不同提示也导致结果波动。本框架可帮助开发者与大模型协同，充分发挥其潜力。
原文摘要 · Abstract (English)
Recent studies have revealed that when LLMs are appropriately prompted and configured, they demonstrate mixed results. Such results often meet or exceed the baseline performance. However, these comparisons have two primary issues. First, they mostly considered only reliability as a comparison metric and selected a few LLMs (such as Codex and ChatGPT) for comparision. This paper proposes a comprehensive code quality assessment framework called Programmatic Excellence via LLM Iteration (PELLI). PELLI is an iterative analysis-based process that upholds high-quality code changes. We extended the state-of-the-art by performing a comprehensive evaluation that generates quantitative metrics for analyzing three primary nonfunctional requirements (such as maintainability, performance, and reliability) while selecting five popular LLMs. For PELLI's applicability, we selected three application domains while following Python coding standards. Following this framework, practitioners can ensure harmonious integration between LLMs and human developers, ensuring that their potential is fully realized. PELLI can serve as a practical guide for developers aiming to leverage LLMs while adhering to recognized quality standards. This study's outcomes are crucial for advancing LLM technologies in real-world applications, providing stakeholders with a clear understanding of where these LLMs excel and where they require further refinement. Overall, based on three nonfunctional requirements, we have found that GPT-4T and Gemini performed slightly better. We also found that prompt design can influence the overall code quality. In addition, each application domain demonstrated high and low scores across various metrics, and even within the same metrics across different prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。