用大模型生成合成数据，低成本验证自然语言生成评估指标。
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
- 用大模型对真实数据进行可控语义降级，生成合成评估数据。
- 多语言问答任务中合成数据与人工评估相关性超0.9，效果可靠。
- 适合缺乏人工标注的场景，或需快速验证指标时使用。
自然语言生成（NLG）评估指标的验证通常依赖昂贵且耗时的人工标注，且主要限于英文数据集。本文提出LLM as a Meta-Judge框架，利用大模型对真实数据进行受控语义降级，生成合成评估数据，替代人工判断。通过元相关性（meta-correlation）衡量合成数据得出的指标排名与标准人工基准的一致性。在机器翻译、问答和摘要任务上的实验表明，合成数据可作为人工判断的可靠代理，在多语言问答任务中元相关性超过0.9，为缺乏人工标注或成本过高场景提供可行替代方案。代码与数据已公开于https://github.com/eiglerl/meta-judge。
原文摘要 · Abstract (English)
Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。