用信息论衡量对话中语义进展,无需大模型也能准确评估。
Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

- 基于嵌入空间的不确定性降低,量化对话中新增的有用信息量。
- 在多个数据集上与人类判断高度一致,且比大模型评分更稳定可靠。
- 轻量级模型即可运行,适合资源受限环境下的对话质量评估。
多轮对话评估困难,因质量体现在多轮交互中而非单轮回复。本文聚焦信息获取型对话的核心维度——语义进展,即对话过程中累积的新颖、相关且非冗余信息。我们将语义进展形式化为条件不确定性的减少,并提出一种基于信息论的嵌入空间近似度量。主估计器采用可计算的高斯公式,具有闭式更新;辅以最大熵论证,说明仅保留二阶嵌入信息时,对数行列式结构自然出现。该方法具备单调性、信息增益可加分解及冗余信息边际递减等理想性质。相比依赖大模型评分的方法,本指标无需生成推理,完全可复现,且在MT-Bench、Chatbot Arena和UltraFeedback上表现优于多个基于LLM的裁判。尤其值得注意的是,即使使用轻量嵌入模型在仅CPU环境下运行,仍保持有效,表明语义进展可脱离大模型能力被捕捉。
原文摘要 · Abstract (English)
Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension of information-seeking dialogue: semantic progress, defined as the accumulation of new, question-relevant, and non-redundant information over the course of a conversation. We formalize semantic progress as question-conditioned uncertainty reduction and introduce an information-theoretic metric that approximates it in embedding space. Our main estimator uses a tractable Gaussian formulation with closed-form updates, while a complementary maximum-entropy argument shows why log-determinant structure arises more broadly when only second-order embedding information is retained. This formulation yields desirable theoretical properties, including monotonicity, additive decomposition of total information gain across turns, and diminishing returns for redundant evidence. Unlike LLM-as-a-judge approaches, our metric requires no autoregressive inference at evaluation time and is fully reproducible for a fixed embedding model. Experiments on MT-Bench, Chatbot Arena, and UltraFeedback show that the proposed metric achieves competitive agreement with human judgments despite targeting only semantic progress, with improved alignment on MT-Bench and UltraFeedback compared to several LLM-based judges. Notably, the method remains effective with lightweight embedding models under CPU-only execution, indicating that semantic progress can be captured without reliance on large model capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。