arXiv:2606.12332cs.CLcs.LG2026-06

用信息论衡量对话中语义进展,无需大模型也能准确评估。

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

论文配图:Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
图 1 · 摘自论文原文
  • 基于嵌入空间的不确定性降低,量化对话中新增的有用信息量。
  • 在多个数据集上与人类判断高度一致,且比大模型评分更稳定可靠。
  • 轻量级模型即可运行,适合资源受限环境下的对话质量评估。

多轮对话评估困难,因质量体现在多轮交互中而非单轮回复。本文聚焦信息获取型对话的核心维度——语义进展,即对话过程中累积的新颖、相关且非冗余信息。我们将语义进展形式化为条件不确定性的减少,并提出一种基于信息论的嵌入空间近似度量。主估计器采用可计算的高斯公式,具有闭式更新;辅以最大熵论证,说明仅保留二阶嵌入信息时,对数行列式结构自然出现。该方法具备单调性、信息增益可加分解及冗余信息边际递减等理想性质。相比依赖大模型评分的方法,本指标无需生成推理,完全可复现,且在MT-Bench、Chatbot Arena和UltraFeedback上表现优于多个基于LLM的裁判。尤其值得注意的是,即使使用轻量嵌入模型在仅CPU环境下运行,仍保持有效,表明语义进展可脱离大模型能力被捕捉。

原文摘要 · Abstract (English)

Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension of information-seeking dialogue: semantic progress, defined as the accumulation of new, question-relevant, and non-redundant information over the course of a conversation. We formalize semantic progress as question-conditioned uncertainty reduction and introduce an information-theoretic metric that approximates it in embedding space. Our main estimator uses a tractable Gaussian formulation with closed-form updates, while a complementary maximum-entropy argument shows why log-determinant structure arises more broadly when only second-order embedding information is retained. This formulation yields desirable theoretical properties, including monotonicity, additive decomposition of total information gain across turns, and diminishing returns for redundant evidence. Unlike LLM-as-a-judge approaches, our metric requires no autoregressive inference at evaluation time and is fully reproducible for a fixed embedding model. Experiments on MT-Bench, Chatbot Arena, and UltraFeedback show that the proposed metric achieves competitive agreement with human judgments despite targeting only semantic progress, with improved alignment on MT-Bench and UltraFeedback compared to several LLM-based judges. Notably, the method remains effective with lightweight embedding models under CPU-only execution, indicating that semantic progress can be captured without reliance on large model capacity.

对话评估信息论轻量模型语义进展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。