将评分标准转化为结构化图，让大模型更精准地打分。
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
- 把评分标准编译成带类型约束的评估图，逻辑更清晰。
- 在四个数据集上精确得分一致率提升0.62至6.75个百分点。
- 适合需要高精度自动评分的研究者和评测系统开发者。
基于评分标准的评估器通常将评分标准视为提示上下文或扁平化指标：它们规定了评判内容，但隐含了指标间的组合关系，即使自然语言规则中已明确说明。我们提出图结构评分标准（GSR），在观察响应前将评分标准编译为与响应无关的、带类型的评估图。准则节点用于获取判断；变换、归约和门控操作通过命名端口组合这些节点；任务特定的输出映射（称为读出）将唯一汇点转换为分数或偏好。编译过程会拒绝非法或类型不兼容的图。点对点评估分别判断各维度，再进行图聚合；成对评估则复用同一图，对每个候选在每项准则下进行一次判断。在GPT-OSS-120B上，GSR在四个点对点数据集上的精确得分一致性比Prometheus风格评分提升0.62至6.75个百分点，并在两个偏好基准上以原生平局和弃权策略实现了最高的端到端成对准确率。
原文摘要 · Abstract (English)
Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。