首次评估大模型对R语言代码智能的支持能力,发现效果显著下降。
Do Current Language Models Support Code Intelligence for R Programming Language?
- 构建首个R语言代码数据集,对比Tidyverse与Base R两种风格
- 多模型在R代码摘要任务中表现普遍下滑,跨项目训练效果差
- 适合关注R语言开发、代码生成或软件工程研究者阅读
近年来,代码预训练语言模型(Code-PLMs)在软件工程领域取得突破性进展,但在流行语言如Java和Python上表现优异的模型,对科学计算语言R的关注却极少。本研究首次系统评估Code-PLMs在R语言上的代码智能能力。我们收集并开源了首个R语言数据集,针对代码摘要与方法名预测两个任务,在不同设置下进行测试,涵盖Tidyverse与Base R两种语法风格。结果表明,现有模型在处理R代码时普遍存在性能退化现象,且经多语言微调后并非所有模型均提升。双重语法范式对代码摘要任务影响尤为显著,同时项目特异性上下文也显著制约跨项目训练效果。
原文摘要 · Abstract (English)
Recent advancements in developing Pre-trained Language Models for Code (Code-PLMs) have urged many areas of Software Engineering (SE) and brought breakthrough results for many SE tasks. Though these models have achieved the state-of-the-art performance for SE tasks for many popular programming languages, such as Java and Python, the Scientific Software and its related languages like R programming language have rarely benefited or even been evaluated with the Code-PLMs. Research has shown that R has many differences with other programming languages and requires specific techniques. In this study, we provide the first insights for code intelligence for R. For this purpose, we collect and open source an R dataset, and evaluate Code-PLMs for the two tasks of code summarization and method name prediction using several settings and strategies, including the differences in two R styles, Tidy-verse and Base R. Our results demonstrate that the studied models have experienced varying degrees of performance degradation when processing R programming language code, which is supported by human evaluation. Additionally, not all models show performance improvement in R-specific tasks even after multi-language fine-tuning. The dual syntax paradigms in R significantly impact the models' performance, particularly in code summarization tasks. Furthermore, the project-specific context inherent in R codebases significantly impacts the performance when attempting cross-project training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。