用语义聚类法评估大模型回答一致性，发现不同方法效果差异明显。
Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answering
- 通过语义聚类识别相同问题的相似回答，避免表面句式差异干扰。
- 在TruthfulQA数据集上，37个问答类别中验证了方法有效性。
- 对比RAG与Zero-shot-CoT，量化分析其对一致性提升的实际效果。
大型语言模型（LLM）在提供可靠信息方面至关重要，但报告指出其输出缺乏一致性。这种不一致常归因于生成过程中的随机性，导致相同问题产生不同回答，削弱用户信任。本文提出一种新的语义一致性评估方法，用于衡量LLM回答的语义连贯性，可识别语法不同但含义相同的句子。以往主要采用两种策略：利用外部知识作为上下文（如RAG模式），或使用Zero-shot-CoT改进模型自身表现。我们在此基础上评估这些方法在不同问答领域对一致性的影响。基于TruthfulQA数据集，对每个问题生成N次响应，并通过语义聚类分析37个类别中的响应一致性。结果定量揭示了这些技术在提升性能前后的实际效果。
原文摘要 · Abstract (English)
In the realm of Large Language Model (LLM) functionalities, providing reliable information is paramount, yet reports suggest that LLM outputs lack consistency. This inconsistency, often at-tributed to randomness in token sampling, under-mines user trust as it leads to varying responses even for identical queries. In this paper, we present a new approach for evaluating semantic consistencies of LLM including comparison of alternative tech-niques. Our approach evaluates whether LLM re-sponses are semantically congruent for a given question, recognizing that as syntactically different sentences may convey the same meaning. Here-tofore, To enhance LLM consistency, two main approaches have been explored: Leverage external knowledge as context like the RAG pattern or use Zero-shot-CoT to improve performance of LLM itself. We apply our evaluation approach to these techniques, and demonstrate to compare the im-pact of these methods on LLM response con-sistency across different domains of question an-swering tasks. Using the TruthfulQA dataset to assess LLM responses, the study induces N re-sponses per question from the LLM and clusters semantically equivalent sentences to measure semantic consistency across 37 categories. Through this, it quantitatively analyzes the effectiveness of the aforementioned methods in improving LLM performance before and after their adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。