用准则对齐方法提升代码偏好判断准确率，解决传统评分方式的不足。
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
- 基于准则的成对判断，直接比较代码质量差异
- 在BigCodeReward上将准确率从60.4%提升至66.3%
- 适合需要可解释性评估代码生成质量的研究者
成对人类偏好预测是评估代码生成系统的核心任务，其质量常取决于功能正确性之外的任务特定权衡。尽管基于评分标准的大型语言模型裁判通过显式分解评价标准提升了可解释性，但现有流程仍多为点对点：独立评分后比较总分。我们指出这种设计与成对偏好预测不匹配，且可能不如强单体裁判。为此提出CriterAlign，一种以准则为中心的框架，通过直接准则级成对判断、由绑定驱动的准则优化、交换一致性过滤和最终成对合成，适配评分标准判断。此外引入离线合成的人类偏好对齐引导（HPAG），从训练样本中提取人类偏好与单体裁判预测间的常见理由差距，并注入准则生成器、准则裁判和最终裁判中。在BigCodeReward数据集上，CriterAlign将Qwen2.5-VL-32B单体裁判准确率从60.4%提升至66.3%，消融实验验证了成对准则设计与HPAG的有效性。
原文摘要 · Abstract (English)
Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。