arXiv:2509.03696cs.IR2025-09中稿 · the CONSEQUENCES W…被引 2

用大模型分析点击数据,精准估算推荐系统的位置偏差。

LLMs for estimating positional bias in logged interaction data

  • 用大模型分析用户点击行为,自动识别位置偏好。
  • 在标准NDCG@10上持平生产模型,加权指标提升约2%。
  • 适合需要低成本评估推荐偏差的工程团队。

推荐与搜索系统通常依赖基于用户日志交互数据训练的排序模型来预测项目相关性。然而,这些交互数据常受位置偏差影响——用户更可能点击排名靠前的项目,而非其真实相关性。因此,新训练的模型可能继承并强化先前模型的偏差,而非真正提升相关性。一种常见缓解方法是逆倾向评分(IPS),即通过倒数权重调整损失函数,该权重为某位置项目被查看概率的估计值。但复杂非线性布局下的倾向估计极具挑战。本文提出一种利用大语言模型(LLMs)分析日志交互数据的新方法,以估算位置偏差。该方法成本低于在线实验。实验表明,使用大模型作为评判者估算的倾向值在得分分桶中稳定,并揭示了Viator网格布局中的行-列效应,而简单启发式方法未能察觉。采用此倾向值进行加权的重排序器,在标准NDCG@10上达到生产模型水平,同时加权NDCG@10提升约2%。后续将通过真实流量实验验证这一离线效果。

原文摘要 · Abstract (English)

Recommender and search systems commonly rely on Learning To Rank models trained on logged user interactions to order items by predicted relevance. However, such interaction data is often subject to position bias, as users are more likely to click on items that appear higher in the ranking, regardless of their actual relevance. As a result, newly trained models may inherit and reinforce the biases of prior ranking models rather than genuinely improving relevance. A standard approach to mitigate position bias is Inverse Propensity Scoring (IPS), where the model's loss is weighted by the inverse of a propensity function, an estimate of the probability that an item at a given position is examined. However, accurate propensity estimation is challenging, especially in interfaces with complex non-linear layouts. In this paper, we propose a novel method for estimating position bias using Large Language Models (LLMs) applied to logged user interaction data. This approach offers a cost-effective alternative to online experimentation. Our experiments show that propensities estimated with our LLM-as-a-judge approach are stable across score buckets and reveal the row-column effects of Viator's grid layout that simpler heuristics overlook. An IPS-weighted reranker trained with these propensities matches the production model on standard NDCG@10 while improving weighted NDCG@10 by roughly 2%. We will verify these offline gains in forthcoming live-traffic experiments.

推荐系统位置偏差大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。