用嵌入距离实现低成本高精度的模型输出自动评估
A-VERT: Agnostic Verification with Embedding Ranking Targets
- 不依赖特定结构，通过语义嵌入距离匹配目标与生成文本
- 在3个数据集上达0.97相关性与96%准确率，接近人工水平
- 适合需要高效评估模型输出质量的研究与工程场景
语言模型输出的自动评估是构建评测基准和度量标准的关键环节，对模型训练和生产端质量评估至关重要。当前方法或成本过高（如大模型作为裁判），或与真实场景差距大（如字符串匹配、对数概率）。本文提出一种无结构评估方法，利用语义嵌入距离将目标候选与任意语言模型生成文本进行匹配，实现低计算开销下的稳健分类（仅需参数少于100亿的嵌入模型）。结果表明，在3个数据集和3种不同语言模型架构上，该方法与人工标注者相比，回归得分约0.97，准确率约96%。
原文摘要 · Abstract (English)
The automatic evaluation of Language Model (LM) responses is a critical piece in the development of benchmarks and metrics, both for model training and quality assessment of production model endpoints. The current approaches to response classification relies on methods that are too expensive (i.e. LLM-as-a-Judge) or that are far from real-world conditions (string-matching, logprob). In this paper, a structure-free evaluation method is presented. The method makes use of semantic embedding distances to match target candidates with arbitrary LM-generated text, resulting in a robust classification of the response at a relatively low compute cost (embedding models of less than $10B$ parameters). The results show a regression score of ~0.97 and an accuracy of ~96% against human annotators, tested over 3 data sets and 3 different LM architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。