arXiv:2410.05193cs.CL2024-10ICLR被引 27

用大模型自动生成适配回复的参考文本,提升自动评估可靠性。

RevisEval: Improving LLM-as-a-Judge via Response-Adapted References

  • 让大模型先改写待评回复,再将其作为参考文本进行评估。
  • 在多种生成任务中超越传统无参考和有参考评估方法。
  • 能降低评估偏差,适合需要高可靠性的生成质量评测场景。

近期研究使大模型作为评判者(LLM-as-a-Judge)成为一种低成本、高效的文本生成质量评估方式。然而,其与人工评估仍存在可靠性差距。一个重要原因是评估过程中缺乏引导性参考。受经典文本评估中参考文本作用的启发,本文提出 RevisEval,一种基于响应自适应参考的新评估范式。核心思想是:理想的参考应与待评估回复保持必要相关性。RevisEval 利用大模型的文本修订能力,先对原始回复进行自适应改写,再将改写后的内容作为参考文本(响应自适应参考)用于后续评估。大量实验表明,RevisEval 在自然语言生成(NLG)任务及开放式指令遵循任务中均优于传统无参考和有参考的评估范式。更重要的是,该方法可进一步提升 BLEU、BERTScore 等经典指标表现,甚至媲美大模型作为评判者。通过详细分析,验证了 RevisEval 在降低偏差、控制推理成本和提升参考相关性方面的有效性。

原文摘要 · Abstract (English)

With significant efforts in recent studies, LLM-as-a-Judge has become a cost-effective alternative to human evaluation for assessing text generation quality in a wide range of tasks. However, there still remains a reliability gap between LLM-as-a-Judge and human evaluation. One important reason is the lack of guided oracles in the evaluation process. Motivated by the role of reference pervasively used in classic text evaluation, we introduce RevisEval, a novel text generation evaluation paradigm via the response-adapted references. RevisEval is driven by the key observation that an ideal reference should maintain the necessary relevance to the response to be evaluated. Specifically, RevisEval leverages the text revision capabilities of large language models (LLMs) to adaptively revise the response, then treat the revised text as the reference (response-adapted reference) for the subsequent evaluation. Extensive experiments demonstrate that RevisEval outperforms traditional reference-free and reference-based evaluation paradigms that use LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks. More importantly, our response-adapted references can further boost the classical text metrics, e.g., BLEU and BERTScore, compared to traditional references and even rival the LLM-as-a-Judge. A detailed analysis is also conducted to confirm RevisEval's effectiveness in bias reduction, the impact of inference cost, and reference relevance.

自动评估大模型评判文本生成参考文本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。