测试代码大模型对设计模式的理解能力，发现其存在显著偏差。
Do Code LLMs Understand Design Patterns?
- 通过识别、理解、生成三方面实验评估模型对设计模式的掌握。
- 模型在识别和生成设计模式时表现不佳，易产生不符合项目规范的代码。
- 适合关注代码生成质量与工程实践适配性的开发者参考。
代码大语言模型（Code LLMs）在代码生成、补全及缺陷检测修复等下游任务中表现出强大适应性。然而，它们常无法捕捉现有编码规范，导致生成的代码与特定项目的既定设计模式冲突，迫使开发人员进行后处理以适配项目标准。本文通过精心设计的实验，从识别、理解到生成三个层面，实证研究了代码大模型在软件开发中的偏见。结果表明，这些偏见显著影响下游任务的可靠性。
原文摘要 · Abstract (English)
Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。