对比开发者与大模型在代码评价中的偏见，发现模型常偏好冗长解释。
Comparing Developer and LLM Biases in Code Evaluation
- 构建框架TRACE，自动提取评分标准并评估模型与人类偏好一致性
- 13个模型在真实编码场景中平均比人工低12-23%的判断准确率
- 发现35处系统性偏差，多数与代码质量标准相关，如偏好长注释
随着大模型越来越多地被用作代码评价裁判，需在包含部分上下文和模糊意图的真实交互场景中对其性能进行评估。我们提出TRACE（代码评价中的量规分析工具），该框架能评估大模型裁判预测人类偏好能力，并自动提取评分维度以揭示人类与模型在权重分配上的系统性偏见。在三种交互模式——基于聊天的编程、IDE自动补全、指令式代码编辑中，使用TRACE测量大模型裁判与开发者偏好的一致性。在13个不同模型中，表现最好的裁判仍比人类标注者低12-23%。TRACE识别出跨模态共35个显著的不一致来源，其中多数对应现有的软件工程代码质量标准。例如，在基于聊天的编程中，模型更偏好较长的代码解释，而人类更倾向简洁表达。结果显示，大多数现有代码质量维度均存在显著偏差，表明大模型裁判与人类偏好在真实应用中仍存在明显差距。
原文摘要 · Abstract (English)
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We present TRACE (Tool for Rubric Analysis in Code Evaluation), a framework that evaluates LLM judges' ability to predict human preferences and automatically extracts rubric items to reveal systematic biases in how humans and models weigh each item. Across three modalities -- chat-based programming, IDE autocompletion, and instructed code editing -- we use TRACE to measure how well LLM judges align with developer preferences. Among 13 different models, the best judges underperform human annotators by 12-23%. TRACE identifies 35 significant sources of misalignment between humans and judges across interaction modalities, the majority of which correspond to existing software engineering code quality criteria. For example, in chat-based coding, judges are biased towards longer code explanations while humans prefer shorter ones. We find significant misalignment on the majority of existing code quality dimensions, showing alignment gaps between LLM judges and human preference in realistic coding applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。