arXiv:2501.12617cs.SEcs.AI2025-01被引 6

现有深度学习方法识别方法名不一致问题,真实场景下效果大幅下降。

Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?

  • 构建新基准:结合提交历史与人工检查,减少误报。
  • 在真实数据上,模型性能显著低于平衡数据集结果。
  • 检索类方法对常见命名有效,生成类方法依赖相似度计算和名称生成质量。

有意义的函数名对代码可读性和维护至关重要,但实际中函数名可能与其实现不一致,导致误解和错误。已有深度学习方法声称能识别此类不一致,但其评估通常基于平衡数据集(不一致与一致样本数量相等),且数据构造存在缺陷,造成大量误报。本文通过结合提交历史自动识别与开发者人工审查,构建新基准,有效降低误报。我们在该基准上评估了五种代表性深度学习方法(1个检索型,4个生成型)。结果表明,从平衡数据集到新基准,性能显著下降。进一步分析发现,检索类方法在简单函数及含常见命名子词的函数上表现良好,但受限于表示能力;生成类方法则因相似度计算不准和名称生成不成熟而表现不佳。基于此,我们提出采用对比学习和大语言模型改进方案。研究指出,当前方法仍无法满足真实软件系统的应用需求。

原文摘要 · Abstract (English)

Concise and meaningful method names are crucial for program comprehension and maintenance. However, method names may become inconsistent with their corresponding implementations, causing confusion and errors. Several deep learning (DL)-based approaches have been proposed to identify such inconsistencies, with initial evaluations showing promising results. However, these evaluations typically use a balanced dataset, where the number of inconsistent and consistent names are equal. This setup, along with flawed dataset construction, leads to false positives, making reported performance less reliable in real-world scenarios, where most method names are consistent. In this paper, we present an empirical study that evaluates state-of-the-art DL-based methods for identifying inconsistent method names. We create a new benchmark by combining automatic identification from commit histories and manual developer inspections, reducing false positives. We evaluate five representative DL approaches (one retrieval-based and four generation-based) on this benchmark. Our results show that performance drops substantially when moving from the balanced dataset to the new benchmark. We further conduct quantitative and qualitative analyses to understand the strengths and weaknesses of the approaches. Retrieval-based methods perform well on simple methods and those with popular name sub-tokens but fail due to inefficient representation techniques. Generation-based methods struggle with inaccurate similarity calculations and immature name generation. Based on these findings, we propose improvements using contrastive learning and large language models (LLMs). Our study suggests that significant improvements are needed before these DL approaches can be effectively applied to real-world software systems.

代码理解深度学习软件工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。