arXiv:2606.22961cs.IR2026-06KDD

用大模型当裁判,让推荐系统评估更可靠、更可解释。

LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation

论文配图:LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation
图 1 · 摘自论文原文
  • 用用户文本行为构建语义偏好代理,避免硬匹配偏差
  • 大模型推理后打分,给出每条推荐的明确理由
  • 适合需要透明评估的推荐系统研发与优化

推荐评估对指导推荐系统优化和部署至关重要。现有方法多依赖离线评估,基于保留用户行为计算Top-K指标,但存在两大根本缺陷:可靠性方面,将观测反馈视为真实偏好并强制硬匹配,而实际反馈受不完整和偏见的物品曝光影响,导致评估失真;可解释性方面,Top-K指标仅提供数值分数,缺乏支撑依据,加剧评估黑箱特性。本文提出一个可靠的、可解释的LLM-as-a-Judge框架。为提升可靠性,引入用户文本行为生成的语义偏好代理,实现偏好与推荐在语义空间中的柔性匹配;为确保可解释性,采用推理-评分流程,由大模型生成相关性判断及显式理由。最终聚合个体评分得到全局Top-K指标,并为每个偏好命中或遗漏提供解释。大量实验表明,该框架在评估可靠性、可解释性和鲁棒性方面表现优异。

原文摘要 · Abstract (English)

Recommendation evaluation plays a crucial role in guiding the refinement and deployment of recommender systems. Most existing trials rely on offline evaluation using Top-K metrics computed over holdout user behaviors. However, we identify two fundamental limitations that undermine their ability to deliver reliable and explainable evaluations. Regarding reliability, offline evaluation treats observed user feedback as a proxy of true preferences and enforces rigid ID matching between the proxy and recommendation. In practice, feedback collections are inherently shaped by incomplete and biased item exposure, leading to distorted and unreliable assessments. Regarding explainability, Top-K metrics only establish numerical scores without offering meaningful insights to support them, thereby reinforcing the black-box nature of offline evaluation. In this paper, we propose a reliable and explainable LLM-as-a-Judge framework for offline recommendation evaluation. To enhance reliability, we introduce a semantic proxy from user textual behaviors to represent their true preferences. This proxy allows for more flexible matching between preferences and recommendations in the semantic space, rather than depending on the holdout feedback. To ensure explainability, the LLM Judge adopts a reasoning-then-scoring process to generate relevance judgments along with explicit rationale. Finally, we aggregate the individual scores into global Top-K metrics to quantify overall recommendation quality, and provide justification for each preference hit or miss. Extensive experiments demonstrate that the LLM Judge achieves solid reliability, explainability, and robustness in evaluation.

推荐系统大模型评估可解释性离线评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。