用分布函数分析大模型生成答案的质量差异，发现相同准确率下表现可能不同。
Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis
- 基于余弦相似度的累积分布函数评估响应质量，超越简单匹配。
- 在问答数据集上区分出准确率相似但分布不同的模型配置。
- 聚类揭示温度、角色、问题主题对回答风格的影响，适合系统调优者参考。
大语言模型（LLMs）被广泛用于复杂任务如问答、科学辩论和软件开发中。标准评估方法将多个模型生成的回答通过多数投票整合为最终答案，并与参考答案对比。然而，这一过程会掩盖原始响应的质量与分布特征。本文提出一种基于生成回答与参考答案余弦相似度经验累积分布函数（ECDF）的新评估框架，实现更细致的质量评估。为分析不同代理配置下的响应分布，我们引入基于距离与k-medoids算法的ECDF聚类方法。在问答数据集上的实验表明，该方法可区分出虽最终准确率相近但质量分布不同的代理设置；聚类分析还揭示了可解释的响应群组结构，反映出温度、角色设定及问题主题的影响。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as agents to solve complex tasks such as question answering (QA), scientific debate, and software development. A standard evaluation procedure aggregates multiple responses from LLM agents into a single final answer, often via majority voting, and compares it against reference answers. However, this process can obscure the quality and distributional characteristics of the original responses. In this paper, we propose a novel evaluation framework based on the empirical cumulative distribution function (ECDF) of cosine similarities between generated responses and reference answers. This enables a more nuanced assessment of response quality beyond exact match metrics. To analyze the response distributions across different agent configurations, we further introduce a clustering method for ECDFs using their distances and the $k$-medoids algorithm. Our experiments on a QA dataset demonstrate that ECDFs can distinguish between agent settings with similar final accuracies but different quality distributions. The clustering analysis also reveals interpretable group structures in the responses, offering insights into the impact of temperature, persona, and question topics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。