用小模型评估对话质量，结合提示词与分类回归模型。
Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations
- 用语言模型提示词做评价，也训练小型分类/回归模型。
- 小模型在部分维度上相关性高，但测试集表现下降。
- 适合资源有限却需多维对话评估的开发者参考。
生成式对话系统日益增多，其评估成为关键挑战。本文参与DSTC-12 Track 1，致力于预测对话级别的多维评分。受限于使用参数少于130亿的小型模型，研究采用两大策略：利用语言模型（LM）通过提示词进行评价，以及训练基于编码器的分类与回归模型。结果显示，尽管LM提示词与人工判断的相关性仅中等，但在测试集上仍位列第二，仅逊于基线模型。分类与回归模型参数显著更少，在验证集上部分维度表现出高相关性；然而其在测试集上性能下降，原因在于测试集某些维度的评分范围与训练及验证集有明显差异。
原文摘要 · Abstract (English)
The growing number of generative AI-based dialogue systems has made their evaluation a crucial challenge. This paper presents our contribution to this important problem through the Dialogue System Technology Challenge (DSTC-12, Track 1), where we developed models to predict dialogue-level, dimension-specific scores. Given the constraint of using relatively small models (i.e. fewer than 13 billion parameters) our work follows two main strategies: employing Language Models (LMs) as evaluators through prompting, and training encoder-based classification and regression models. Our results show that while LM prompting achieves only modest correlations with human judgments, it still ranks second on the test set, outperformed only by the baseline. The regression and classification models, with significantly fewer parameters, demonstrate high correlation for some dimensions on the validation set. Although their performance decreases on the test set, it is important to note that the test set contains annotations with significantly different score ranges for some of the dimensions with respect to the train and validation sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。