用LLM当裁判分析模型错误原因，让评估不只是打分。
CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
- 用LLM生成每条数据的错误反馈，提炼系统级问题
- 量化各类错误出现频率，揭示模型短板
- 交互式仪表板支持深入追踪具体错误案例
大型语言模型（LLMs）的评估越来越多依赖其他LLM作为评判者。然而，现有评估范式通常只给出单一分数或排名，回答‘哪个模型更好’，却无法解释‘为什么’。这类顶层评分掩盖了模型表现背后的具体、可操作的原因。为弥补这一差距，我们提出CLEAR——一个用于基于LLM的错误分析的交互式开源工具包。CLEAR首先生成每个实例的文本反馈，再归纳出系统级错误类别，并量化每类问题的普遍程度。该工具包还提供交互式仪表板，支持通过聚合可视化进行综合错误分析，利用交互式过滤器定位特定问题或得分区间，并深入查看体现特定行为模式的个体实例。我们在RAG和数学推理基准上展示了CLEAR的分析效果，并通过用户案例研究验证其实用性。
原文摘要 · Abstract (English)
The evaluation of Large Language Models (LLMs) increasingly relies on other LLMs acting as judges. However, current evaluation paradigms typically yield a single score or ranking, answering which model is better but not why. While essential for benchmarking, these top-level scores obscure the specific, actionable reasons behind a model's performance. To bridge this gap, we introduce CLEAR, an interactive, open-source package for LLM-based error analysis. CLEAR first generates per-instance textual feedback, then it creates a set of system-level error issues, and quantifies the prevalence of each identified issue. Our package also provides users with an interactive dashboard that allows for a comprehensive error analysis through aggregate visualizations, applies interactive filters to isolate specific issues or score ranges, and drills down to the individual instances that exemplify a particular behavioral pattern. We demonstrate CLEAR analysis for RAG and Math benchmarks, and showcase its utility through a user case study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。