用微调大模型生成文本相关性标签，提升应用商店搜索排名效果
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
- 用细调大模型替代预训练模型生成更精准的文本相关性标签
- 生成数百万标签后，离线评估中行为与文本相关性均显著提升
- 在长尾查询中表现突出，适合解决低行为数据场景的推荐问题
大型商业搜索系统为提升用户会话成功率，需优化结果的相关性。为此，我们结合行为相关性（用户点击或下载倾向）与文本相关性（结果与查询语义匹配度）。由于专家标注的文本相关性数据远少于行为数据，我们系统评估了多种大模型配置，发现一个经过专门微调的模型显著优于更大规模的预训练模型。利用该最优模型作为放大器，我们生成了数百万条文本相关性标签，缓解数据稀缺问题。实验表明，将这些标签融入生产排序器后，帕累托前沿向外显著移动：离线NDCG在行为相关性不变的情况下，文本相关性大幅提升。全球范围的A/B测试验证了这一改进，整体转化率提升0.24%，尤其在长尾查询中效果最明显，此时文本相关性标签提供了可靠信号。
原文摘要 · Abstract (English)
Large-scale commercial search systems optimize for relevance to drive successful sessions that help users find what they are looking for. To maximize relevance, we leverage two complementary objectives: behavioral relevance (results users tend to click or download) and textual relevance (a result's semantic fit to the query). A persistent challenge is the scarcity of expert-provided textual relevance labels relative to abundant behavioral relevance labels. We first address this by systematically evaluating LLM configurations, finding that a specialized, fine-tuned model significantly outperforms a much larger pre-trained one in providing highly relevant labels. Using this optimal model as a force multiplier, we generate millions of textual relevance labels to overcome the data scarcity. We show that augmenting our production ranker with these textual relevance labels leads to a significant outward shift of the Pareto frontier: offline NDCG improves for behavioral relevance while simultaneously increasing for textual relevance. These offline gains were validated by a worldwide A/B test on the App Store ranker, which demonstrated a statistically significant +0.24% increase in conversion rate, with the most substantial performance gains occurring in tail queries, where the new textual relevance labels provide a robust signal in the absence of reliable behavioral relevance labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。