用大模型标注数据构建电商搜索召回系统,摆脱点击依赖。
Scaling Dense Retrieval with LLM-Annotated Training Data: Structured Mining and Progressive Curriculum for E-Commerce Sponsored Search
- 三系统交叉检索生成结构化训练信号,自动标注正负样本。
- 240万+样本分五级难度训练,尾部查询召回提升5.1%。
- 线上测试广告花费增2.8%,点击率和转化率显著提高。
如何在不依赖点击数据或人工标注的前提下,为稠密检索模型生成高质量训练数据?这对电商广告搜索至关重要——点击训练存在位置偏差与长尾查询稀疏问题,而百万级查询-商品对的人工标注成本过高。本文发现:不同检索系统在召回结果上存在广泛分歧,这种分歧可作为天然的结构化信号:一致召回为易正例,仅词法系统识别的为难正例,仅单个系统误判的为难负例。为此提出端到端方案:(a)融合三个生产系统带排名元信息的多通道检索挖掘;(b)基于校准三模型级联的分级相关性标注,与人工标注达成89.1%一致率;(c)三阶段渐进式课程学习,组织2400万+训练样本跨五个难度层级。在沃尔玛广告搜索中部署两塔BERT模型,以3万条经第三方专家标注的查询进行评估。结果表明:相较点击训练基线,NDCG@10提升5.1%,尾部查询收益最大;令人尴尬的召回(评分0)从8.7%降至3.5%。两周线上A/B测试显示,每臂处理数千万次广告请求,广告支出+2.80%,点击率+1.4%,每千次展示收益(eCPM)+2.8%,点击转化率+2.9%。整体证明,该方法可实用、规模化替代点击驱动训练,实现基于大模型标注的监督学习。
原文摘要 · Abstract (English)
How can we generate high-quality training data for dense retrieval models at production scale, without relying on click signals or manual annotation? This question is critical for e-commerce sponsored search, where click-based training suffers from position bias and tail-query sparsity, and manual labeling at the scale of hundreds of millions of query-item pairs is economically infeasible. Our work is driven by the following insight: heterogeneous retrieval systems disagree on most items they retrieve, and this disagreement creates a natural source of structured training signal -- easy positives where all systems agree, hard positives that only lexical systems find, and hard negatives that fool exactly one system. As our key novelty, we combine three ideas into an end-to-end pipeline: (a) multi-channel retrieval mining with rank metadata from three production systems, (b) graded-relevance annotation by a calibrated three-model cascade ) that reaches 89.1% agreement with trained human annotators, and (c) three-stage progressive curriculum training that organizes 240M+ training examples across five difficulty levels. We deploy the trained two-tower BERT model on Walmart's sponsored search and evaluate it against 30K queries labeled by trained third-party human annotators. First, we show that the system achieves +5.1% NDCG@10 over the click-trained production baseline, with the largest gain on tail queries . Second, we show that embarrassing retrievals (rating 0) drop from 8.7% to 3.5%. Third, a two-week online A/B test with tens of millions of ad requests per arm confirms +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate. Overall, our work provides a practical and scalable blueprint for replacing click-based training with structured LLM-annotated supervision in production retrieval systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。