提出评估代码生成模型的综合指标InI，更全面衡量效率、一致性和准确率。
Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index
- 构建包含效率、一致性、准确率的三维度评估指标InI
- GPT-4o在准确率和效率上优于OAI1，接近OAI3
- 有效提示与迭代可让LLM生成媲美专家的代码
本研究提出一种新的推理指数（InI）评估方法——INFINITE（INFerence INdex In Testing model Effectiveness methodology），用于评估大语言模型（LLMs）在代码生成任务中的表现。该指数从效率、一致性与准确性三个关键维度进行综合评估，涵盖时间效率、响应质量及输出稳定性，突破传统仅依赖准确率的评价局限。研究将该方法应用于比较OpenAI的GPT-4o（GPT）、OpenAI-o1 pro（OAI1）和OpenAI-o3 mini-high（OAI3）在生成用于预测温度、相对湿度和风速等气象变量的LSTM模型Python代码的表现。结果表明，GPT在准确率和工作流效率上优于OAI1，且与OAI3表现相当。研究还发现，通过有效提示与迭代优化，LLM生成的代码可达到专家设计水平。GPT的性能优势凸显了其广泛使用与用户反馈带来的提升价值。
原文摘要 · Abstract (English)
This study introduces a new methodology for an Inference Index (InI), called INFerence INdex In Testing model Effectiveness methodology (INFINITE), aiming to evaluate the performance of Large Language Models (LLMs) in code generation tasks. The InI index provides a comprehensive assessment focusing on three key components: efficiency, consistency, and accuracy. This approach encapsulates time-based efficiency, response quality, and the stability of model outputs, offering a thorough understanding of LLM performance beyond traditional accuracy metrics. We applied this methodology to compare OpenAI's GPT-4o (GPT), OpenAI-o1 pro (OAI1), and OpenAI-o3 mini-high (OAI3) in generating Python code for the Long-Short-Term-Memory (LSTM) model to forecast meteorological variables such as temperature, relative humidity and wind velocity. Our findings demonstrate that GPT outperforms OAI1 and performs comparably to OAI3 regarding accuracy and workflow efficiency. The study reveals that LLM-assisted code generation can produce results similar to expert-designed models with effective prompting and refinement. GPT's performance advantage highlights the benefits of widespread use and user feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。