用专家提示提升大模型代码评估可靠性,显著减少盲区漏检。
Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls
- 构建领域特定检查器,动态注入提示以引导大模型重审遗漏点。
- 结合提示增强后,错误检测率从45%-63%提升至74%。
- 适合关注工业级代码生成评估可靠性的研发与测试人员。
大语言模型作为代码生成评估裁判(LaaJ)虽具可扩展性,但在关键任务中常忽略领域特异性问题。本文以企业级COBOL代码现代化为案例,发现即使上线部署的LaaJ仍会持续遗漏关键错误。通过专家知识构建初步分类体系,开发轻量级分析检查器,可识别30余种实际场景中的领域问题。将检查器输出作为提示注入评估流程,实验显示仅靠LaaJ检测准确率仅为45%-63%,而分析提示组合使最佳模型检测覆盖率提升至74%,且解释更精准丰富。结果表明,分析-大模型混合方法能显著提升评估可靠性。数据集与所有提示已公开。
原文摘要 · Abstract (English)
Large Language Models are increasingly deployed as judges (LaaJ) in code generation pipelines. While attractive for scalability, LaaJs tend to overlook domain specific issues raising concerns about their reliability in critical evaluation tasks. To better understand these limitations in practice, we examine LaaJ behavior in a concrete industrial use case: legacy code modernization via COBOL code generation. In this setting, we find that even production deployed LaaJs can miss domain critical errors, revealing consistent blind spots in their evaluation capabilities. To better understand these blind spots, we analyze generated COBOL programs and associated LaaJs judgments, drawing on expert knowledge to construct a preliminary taxonomy. Based on this taxonomy, we develop a lightweight analytic checker tool that flags over 30 domain specific issues observed in practice. We use its outputs as analytic hints, dynamically injecting them into the judges prompt to encourage LaaJ to revisit aspects it may have overlooked. Experiments on a test set of 100 programs using four production level LaaJs show that LaaJ alone detects only about 45-63% of the errors present in the code (in all judges we tested), while the analytic checker alone lacks explanatory depth. When combined, the LaaJ+Hints configuration achieves up to 74% coverage (for the best performing judge and injection prompt) and produces qualitatively richer, more accurate explanations, demonstrating that analytic-LLM hybrids can substantially enhance evaluation reliability in deployed pipelines. We release the dataset and all used prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。