用大模型判题可信吗?实测发现它常误判难度,反而不如传统算法靠谱。
Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks
- 对比大模型与轻量机器学习模型对编程题难度的判断能力
- 大模型准确率仅37.75%,远低于轻量模型86%的精度
- 大模型忽视输入规模等关键指标,适合教育平台谨慎使用
大型语言模型(LLMs)在自然语言和代码生成方面表现优异,正被广泛用于自动评估模型输出与学习活动。然而其在结构化任务如预测竞赛编程题难度方面的表现仍缺乏深入研究。我们在包含1,825道LeetCode题目(标注为易、中、难)的数据集上,系统比较了仅以自然语言评估的GPT-4o与基于显式数值和文本特征训练的可解释LightGBM集成模型。结果表明,LightGBM达到86%准确率,而GPT-4o仅为37.75%。详细分析(包括混淆矩阵与SHAP可解释性)显示,输入规模限制和通过率等数值约束是区分难题与简单题的关键因素,但GPT-4o常忽略这些线索,并表现出对简单类别的强烈偏倚。我们进一步通过合成难题生成协议测试GPT-4o:令人惊讶的是,它几乎将自身生成的所有难题标记为中等,与其将真实难题降级为简单的倾向形成反差。这些发现揭示了大模型作为裁判的明确失效模式,强调在竞赛编程、教育平台或强化学习流程中采用前必须解决这些问题。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities in natural language and code generation, and are increasingly deployed as automatic judges of model outputs and learning activities. Yet, their behavior on structured tasks such as predicting the difficulty of competitive programming problems remains under-explored. We conduct a systematic comparison of GPT-4o, used purely as a natural-language difficulty assessor, against an interpretable Light-GBM ensemble trained on explicit numeric and textual features. On a dataset of 1,825 LeetCode problems labeled Easy, Medium, or Hard, LightGBM attains 86% accuracy, whereas GPT-4o reaches only 37.75%. Detailed analyses, including confusion matrices and SHAP-based interpretability, show that numeric constraints -- such as input size limits and acceptance rates -- play a crucial role in separating Hard problems from easier ones. By contrast, GPT-4o often overlooks these cues and exhibits a strong bias toward simpler categories. We further probe GPT-4o through a synthetic Hard-problem generation protocol. Surprisingly, GPT-4o labels almost all of its own synthetic Hard problems as Medium, contradicting its tendency to downgrade real Hard problems to Easy. Our findings connect to recent work on LLMs-as-judges and automatic difficulty estimation in programming and education, and highlight concrete failure modes that must be addressed before LLM-based judges can be considered trustworthy in competitive programming, educational platforms, or reinforcement-learning pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。