发现大模型评估文本时会受无关维度干扰，提出新方法提升评估准确性。
Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text
- 通过相关性差异量化大模型评估中的跨维度依赖问题。
- 提出DimCheck方法，迭代剔除无关推理证据，显著降低依赖性。
- 小模型用该方法可逼近大模型效果，推理成本更低，适合实用部署。
LLM-as-a-judge方法广泛用于生成式开放文本的质量评估。由于不同维度的错误模式各异，可靠评估需独立处理各维度。本文提出CorrGap，通过比较不同文本组中模型预测分与真实分的相关性差异，量化大模型在评估某一维度时对非目标维度的依赖程度。实验表明，跨维度依赖在各类大模型中普遍存在。为此，我们提出DimCheck，通过逐步移除链式推理（COT）中与目标维度无关的证据，有效缓解依赖问题。在三个大模型和四个任务上的结果表明，DimCheck优于多个强基线。此外，较小的训练模型在使用DimCheck时可逼近大模型性能，同时显著降低推理开销。
原文摘要 · Abstract (English)
LLM-as-a-judge methods are widely used for evaluating the quality of generated open-ended text. Such evaluations are generally multi-dimensional, since the error patterns in texts can be different for different dimensions. Therefore, reliable LLM judges should evaluate each target dimension independently. To quantify the extent to which LLM judges depend on non-target dimensions when evaluating a target dimension, i.e., inter-dimension dependence, we propose CorrGap. To measure this, CorrGap uses the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts. Using CorrGap, we show that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks. To mitigate inter-dimension dependence, we propose DimCheck, a method that iteratively removes unrelated evidence from COTs generated by LLM judges in a step-wise way. We show that DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks. We also show that smaller trained LLMs can approximate larger LLMs in DimCheck, with much lower inference costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。