对比多个开源大模型在代码摘要任务中的表现
Analysis on LLMs Performance for Code Summarization
- 测试 LLaMA-3、Phi-3、Mistral、Gemma 四个模型的代码摘要能力
- 使用 BLEU₃.₁ 和 ROUGE₃.₂ 评估，发现模型间性能差异明显
- 适合关注代码自动化工具研发的研究者与开发者参考
代码摘要旨在为源代码生成简洁的自然语言描述。近年来，深度学习在软件工程中广泛应用，尤其在代码生成与摘要任务中表现突出。当前大型语言模型（LLMs）在编码任务上展现出良好性能。本研究对 LLaMA-3、Phi-3、Mistral、Gemma 等开源大模型进行对比分析，采用 BLEU₃.₁ 与 ROUGE₃.₂ 等关键指标评估其在代码摘要任务中的表现。通过该分析，揭示各模型的优势与局限，为它们在代码摘要中的适用性与有效性提供见解。研究成果有助于推动 LLMs 的持续优化，并支持其在提升软件开发与维护效率的工具中集成。
原文摘要 · Abstract (English)
Code summarization aims to generate concise natural language descriptions for source code. Deep learning has been used more and more recently in software engineering, particularly for tasks like code creation and summarization. Specifically, it appears that the most current Large Language Models with coding perform well on these tasks. Large Language Models (LLMs) have significantly advanced the field of code summarization, providing sophisticated methods for generating concise and accurate summaries of source code. This study aims to perform a comparative analysis of several open-source LLMs, namely LLaMA-3, Phi-3, Mistral, and Gemma. These models' performance is assessed using important metrics such as BLEU\textsubscript{3.1} and ROUGE\textsubscript{3.2}. Through this analysis, we seek to identify the strengths and weaknesses of each model, offering insights into their applicability and effectiveness in code summarization tasks. Our findings contribute to the ongoing development and refinement of LLMs, supporting their integration into tools that enhance software development and maintenance processes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。