无需标注数据，用对话概率衡量聊天投入度
PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement
- 基于互信息思想，用上下文生成回复的概率衡量参与度
- 在真实和合成数据上验证，能有效估计对话参与度
- 适合评估大模型对话质量，尤其无标签场景
高对话参与度是有效交流的关键指标。可靠的参与度度量可助力大语言模型评测、提升人机交互效果或改善个人沟通能力。然而，参与度主观性强且缺乏标准参考，量化困难。本文提出PMIScore，一种高效的无监督对话参与度量化方法。该方法利用点互信息（PMI），即给定对话历史时生成回复的概率。由于直接计算PMI不可行，PMIScore通过双形式分歧学习实现。算法包括生成正负对话对、用大语言模型提取嵌入表示，并使用互信息损失函数训练小型神经网络。我们在合成与真实数据集上验证了PMIScore的有效性，结果表明其在PMI估计方面表现良好，且所提的PMI度量具有合理性。
原文摘要 · Abstract (English)
High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。