提升代码摘要质量,引入类与仓库上下文增强理解
Code Summarization Beyond Function Level
- 在函数级之外引入类和仓库上下文进行代码摘要
- 使用RAG和少样本学习使大模型在类/仓库级别性能显著提升
- 适合关注代码理解、提示工程与检索增强的开发者
代码摘要在自然语言处理与软件工程中至关重要,旨在生成源代码的简洁描述。近期进展提升了摘要质量,增强了代码可读性与可维护性。然而,现有研究未考虑仓库或类级别的内容。本研究探索了函数级以外的代码摘要效果,分析类与仓库上下文对摘要质量的影响。通过重构类与仓库级别的评估基准,评估基线模型,并利用上下文学习测试大模型表现。结果表明,微调后的CodeT5+ base模型表现最优;结合少样本学习与RAG检索代码片段,显著提升了LLMs在类与仓库层级的表现。Deepseek Coder 1.3B与Starcoder2 15B在BLEURT、METEOR、BLEU-4等指标上均有显著提升。仓库级摘要潜力巨大,但需大量计算资源,并受益于结构化上下文。研究采用最新SIDE评估指标,推动提示工程、少样本学习与RAG策略优化,填补多层级代码摘要评估空白。所有数据、代码与结果已公开于GitHub:https://github.com/kilimanj4r0/code-summarization-beyond-function-level。
原文摘要 · Abstract (English)
Code summarization is a critical task in natural language processing and software engineering, which aims to generate concise descriptions of source code. Recent advancements have improved the quality of these summaries, enhancing code readability and maintainability. However, the content of a repository or a class has not been considered in function code summarization. This study investigated the effectiveness of code summarization models beyond the function level, exploring the impact of class and repository contexts on the summary quality. The study involved revising benchmarks for evaluating models at class and repository levels, assessing baseline models, and evaluating LLMs with in-context learning to determine the enhancement of summary quality with additional context. The findings revealed that the fine-tuned state-of-the-art CodeT5+ base model excelled in code summarization, while incorporating few-shot learning and retrieved code chunks from RAG significantly enhanced the performance of LLMs in this task. Notably, the Deepseek Coder 1.3B and Starcoder2 15B models demonstrated substantial improvements in metrics such as BLEURT, METEOR, and BLEU-4 at both class and repository levels. Repository-level summarization exhibited promising potential but necessitates significant computational resources and gains from the inclusion of structured context. Lastly, we employed the recent SIDE code summarization metric in our evaluation. This study contributes to refining strategies for prompt engineering, few-shot learning, and RAG, addressing gaps in benchmarks for code summarization at various levels. Finally, we publish all study details, code, datasets, and results of evaluation in the GitHub repository available at https://github.com/kilimanj4r0/code-summarization-beyond-function-level.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。