用大模型自动评估文本一致性并给出解释,效果优于现有方法。
AXCEL: Automated eXplainable Consistency Evaluation using LLMs
- 基于提示的评估方法,无需改写提示即可跨任务通用。
- 在摘要生成、自由文本生成等任务中检测不一致率提升6.2%至29.4%。
- 输出带推理过程和具体矛盾段落,适合需要可解释性的研究与应用。
大语言模型在工业与学术界广泛应用,但生成文本的一致性评估仍具挑战。传统指标如ROUGE、BLEU与人工判断相关性弱;基于自然语言推理(NLI)的指标虽有改进,但实现复杂、需领域定制训练且缺乏可解释性。近期基于提示的评估方法虽易部署,但仍依赖特定任务提示,可解释性不足。本文提出AXCEL:一种基于提示的一致性评估方法,能提供评分依据的详细推理,并精确定位不一致文本片段。该方法具备良好泛化能力,适用于多种任务而无需修改提示。在摘要生成任务中性能优于当前最先进方法8.7%,自由文本生成任务提升6.2%,数据到文本转换任务提升29.4%。我们还评估了底层大模型对提示类指标的影响,并使用最新大模型重新校准了现有方法以保证公平比较。进一步验证显示,AXCEL在开源大模型上亦表现优异。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely used in both industry and academia for various tasks, yet evaluating the consistency of generated text responses continues to be a challenge. Traditional metrics like ROUGE and BLEU show a weak correlation with human judgment. More sophisticated metrics using Natural Language Inference (NLI) have shown improved correlations but are complex to implement, require domain-specific training due to poor cross-domain generalization, and lack explainability. More recently, prompt-based metrics using LLMs as evaluators have emerged; while they are easier to implement, they still lack explainability and depend on task-specific prompts, which limits their generalizability. This work introduces Automated eXplainable Consistency Evaluation using LLMs (AXCEL), a prompt-based consistency metric which offers explanations for the consistency scores by providing detailed reasoning and pinpointing inconsistent text spans. AXCEL is also a generalizable metric which can be adopted to multiple tasks without changing the prompt. AXCEL outperforms both non-prompt and prompt-based state-of-the-art (SOTA) metrics in detecting inconsistencies across summarization by 8.7%, free text generation by 6.2%, and data-to-text conversion tasks by 29.4%. We also evaluate the influence of underlying LLMs on prompt based metric performance and recalibrate the SOTA prompt-based metrics with the latest LLMs for fair comparison. Further, we show that AXCEL demonstrates strong performance using open source LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。