arXiv:2512.22629cs.AIcs.IR2025-12中稿 · NeurIPS

DICE让RAG系统评估更可信:用概率打分+锦标赛算法,结果可解释且高效。

DICE: Discrete Interpretable Comparative Evaluation with Probabilistic Scoring for Retrieval-Augmented Generation

  • 采用双阶段框架,结合证据链与概率评分(A/B/平局),实现可解释判断
  • 在8个系统对比中计算量降低42.9%,排名准确率保持不变
  • 中文金融问答数据集上与专家一致率达85.7%,优于现有评测方法

随着检索增强生成(RAG)系统架构日益复杂,其可信性评估亟需具备可解释性与鲁棒性。现有标量指标存在可解释性差、不确定性量化不足、多系统比较效率低等问题,制约了RAG技术的负责任部署。本文提出DICE(Discrete Interpretable Comparative Evaluation),一种两阶段、证据耦合的评估框架,通过深度分析推理与概率{A, B, Tie}评分,生成透明且带置信度的判断,支持可追溯的推理过程,助力系统改进与错误诊断。为提升大规模评估效率,DICE采用瑞士轮锦标赛机制,将计算复杂度从$O(N^2)$降至$O(N \log N)$,在八系统评估中实现42.9%的计算量下降,同时保持排名一致性。在自建中文金融问答数据集上的验证表明,DICE与人类专家达成85.7%的一致性,显著优于现有基于LLM的指标如RAGAS。结果确立了DICE作为可信、可解释、高效的RAG系统评估新范式。

原文摘要 · Abstract (English)

As Retrieval-Augmented Generation (RAG) systems evolve toward more sophisticated architectures, ensuring their trustworthiness through explainable and robust evaluation becomes critical. Existing scalar metrics suffer from limited interpretability, inadequate uncertainty quantification, and computational inefficiency in multi-system comparisons, hindering responsible deployment of RAG technologies. We introduce DICE (Discrete Interpretable Comparative Evaluation), a two-stage, evidence-coupled framework that advances explainability and robustness in RAG evaluation. DICE combines deep analytical reasoning with probabilistic $\{A, B, Tie\}$ scoring to produce transparent, confidence-aware judgments that support accountable system improvement through interpretable reasoning traces, enabling systematic error diagnosis and actionable insights. To address efficiency challenges at scale, DICE employs a Swiss-system tournament that reduces computational complexity from $O(N^2)$ to $O(N \log N)$, achieving a 42.9% reduction in our eight-system evaluation while preserving ranking fidelity. Validation on a curated Chinese financial QA dataset demonstrates that DICE achieves 85.7% agreement with human experts, substantially outperforming existing LLM-based metrics such as RAGAS. Our results establish DICE as a responsible, explainable, and efficient paradigm for trustworthy RAG system assessment.

RAG评估可解释性概率评分高效计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。