用大模型自动分析文本生成错误类型，帮开发者精准改进系统。
LLM-as-a-qualitative-judge: automating error analysis in natural language generation
- 让大模型输出结构化错误报告，而非单一分数。
- 在12个数据集上，80%的错误识别与人工标注一致。
- 适合需要深度调试生成系统的开发团队使用。
将大语言模型（LLM）用于文本生成评估，传统做法是输出数值评分，但本工作提出一种新方法——以定性报告为主输出的LLM-as-a-qualitative-judge。该方法分两步：先对每条生成结果进行开放式的错误分析，再通过直观的累积算法聚类发现的问题。研究引入了约300条来自12个文本生成数据集的错误标注，验证其有效性。结果显示，该方法在2/3情况下能准确匹配人工标注的实例级错误；同时生成的错误类型报告与人类撰写报告高度相似。案例研究进一步证明，利用该方法可显著提升文本生成系统性能。代码与数据已公开于GitHub。
原文摘要 · Abstract (English)
Prompting large language models (LLMs) to evaluate generated text, known as LLM-as-a-judge, has become a standard evaluation approach in natural language generation (NLG), but is primarily used as a quantitative tool, i.e. with numerical scores as main outputs. In this work, we propose LLM-as-a-qualitative-judge, an LLM-based evaluation approach with the main output being a structured report of common issue types in the NLG system outputs. Our approach is targeted at providing developers with meaningful insights on what improvements can be done to a given NLG system and consists of two main steps, namely open-ended per-instance issue analysis and clustering of the discovered issues using an intuitive cumulative algorithm. We also introduce a strategy for evaluating the proposed approach, coupled with ~300 annotations of issues in instances from 12 NLG datasets. Our results show that instance-specific issues output by LLM-as-a-qualitative-judge match those annotated by humans in 2/3 cases, and that LLM-as-a-qualitative-judge is capable of producing error type reports resembling the reports composed by human annotators. We also demonstrate in a case study how the use of LLM-as-a-qualitative-judge can substantially improve NLG systems performance. Our code and data are publicly available at https://github.com/tunde-ajayi/llm-as-a-qualitative-judge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。