用提示工程让大模型检测多语言代码的SOLID设计缺陷，效果因模型和提示策略而异。
Are We SOLID Yet? An Empirical Study on Prompting LLMs to Detect Design Principle Violations
- 设计专用提示词，评估大模型跨语言识别五大SOLID原则违规的能力。
- GPT-4o Mini表现最优，但对依赖倒置原则（DIP）仍难应对，准确率不足60%。
- 不同提示策略效果差异大，需根据具体设计问题匹配模型与提示组合。
传统静态分析难以发现语义层面的设计缺陷，如违反SOLID原则，这类问题需要深入理解面向对象设计模式。现有方法多聚焦单一原则或特定语言，缺乏对多语言代码库中全部五项原则的统一检测能力。本文提出一种新方法：通过定制化提示工程，评估大模型在多语言环境下检测SOLID违规的能力。我们构建了一个包含240个手动验证代码片段的新基准数据集，并测试了四种受零样本、少样本和思维链启发的提示策略，系统评估其对检测准确率的影响。结果表明，模型间性能差距显著，GPT-4o Mini表现最佳，但在依赖倒置原则（DIP）上仍面临挑战。提示策略影响巨大，无通用最优方案——例如，推理性集成提示在开闭原则（OCP）上表现优异，而提示引导示例策略在DIP检测中更优。所有实验均显示，检测准确率受语言特性影响明显，且随代码复杂度上升急剧下降。研究说明，有效的AI驱动设计分析并非依赖单一模型，而是需根据具体场景匹配模型与提示策略，凸显大模型在提升代码可维护性方面的潜力。
原文摘要 · Abstract (English)
Traditional static analysis methods struggle to detect semantic design flaws, such as violations of the SOLID principles, which require a strong understanding of object-oriented design patterns and principles. Existing solutions typically focus on individual SOLID principles or specific programming languages, leaving a gap in the ability to detect violations across all five principles in multi-language codebases. This paper presents a new approach: a methodology that leverages tailored prompt engineering to assess LLMs on their ability to detect SOLID violations across multiple languages. We present a benchmark of four leading LLMs-CodeLlama, DeepSeekCoder, QwenCoder, and GPT-4o Mini-on their ability to detect violations of all five SOLID principles. For this evaluation, we construct a new benchmark dataset of 240 manually validated code examples. Using this dataset, we test four distinct prompt strategies inspired by established zero-shot, few-shot, and chain-of-thought techniques to systematically measure their impact on detection accuracy. Our emerging results reveal a stark hierarchy among models, with GPT-4o Mini decisively outperforming others, yet even struggles with challenging principles like DIP. Crucially, we show that prompt strategy has a dramatic impact, but no single strategy is universally best; for instance, a deliberative ENSEMBLE prompt excels at OCP detection while a hint-based EXAMPLE prompt is superior for DIP violations. Across all experiments, detection accuracy is heavily influenced by language characteristics and degrades sharply with increasing code complexity. These initial findings demonstrate that effective, AI-driven design analysis requires not a single best model, but a tailored approach that matches the right model and prompt to the specific design context, highlighting the potential of LLMs to support maintainability through AI-assisted code analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。