arXiv:2606.27226cs.AIcs.CL2026-06被引 1

用二元问题分解评估,让大模型评价更透明可调试。

Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

论文配图:Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
图 1 · 摘自论文原文
  • 将评价拆成细粒度二元问题,由大模型独立回答
  • 在事实一致性任务上超越基线,更贴近人类评分分布
  • 反馈可直接用于提示词优化,适合研究者调试模型

大模型输出评估仍是自然语言处理的瓶颈:人工评估成本高,词法指标与人类判断相关性差,全貌式大模型评判常产生难以调试的黑箱分数。本文提出BINEVAL框架,将评估标准分解为原子级二元问题,通过元提示生成细粒度提问,由大模型独立回答每个输出,获得可解释的问题级反馈及校准后的综合评分。该方法使评估更易检查、诊断,且可直接用于提示优化。在SummEval、Topical-Chat和QAGS等数据集上,BINEVAL表现优于或相当强基线(如UniEval、G-Eval),尤其在事实一致性任务(如QAGS)上优势显著。其评分与人类判断相关性强,更贴近人类评分分布,避免了以往模型评判的天花板效应,更好区分边缘与明显错误输出。进一步证明,相同问题级反馈支持迭代提示优化,在总结与生成任务上,可在自更新与跨模型更新设置下提升评测提示与生成提示性能。整体上,BINEVAL提供一种无需训练、任务无关、可解释的评估框架,兼具优异实证表现与实用诊断优化价值。

原文摘要 · Abstract (English)

Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement. Across SummEval, Topical-Chat, and QAGS, BINEVAL matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BINEVAL better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BINEVAL provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.

模型评估可解释性提示优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。