用强化学习提升大模型生成搜索查询的精准度
Token-level Proximal Policy Optimization for Query Generation
- 在词元级别设计奖励模型与策略优化，解决反馈稀疏问题
- 在开源与工业数据集上显著优于现有方法
- 适合需要精准意图理解的搜索与推荐系统开发者
查询生成是搜索引擎（如 Google、Bing）和推荐系统的关键任务。近期先进方法利用大语言模型（LLMs）强大的上下文理解与文本生成能力，但仍难以基于用户搜索历史准确推断其意图。本文提出词元级近端策略优化（TPPO），一种基于人工智能反馈强化学习（RLAIF）范式的新型微调方法。TPPO包含词元级奖励模型与词元级近端策略优化模块，有效应对传统RLAIF框架中的奖励稀疏问题。我们在开源数据集和来自全球使用搜索引擎的工业数据集上进行了实验，结果表明TPPO显著提升了LLMs在查询生成上的表现，优于现有竞争方法。
原文摘要 · Abstract (English)
Query generation is a critical task for web search engines (e.g. Google, Bing) and recommendation systems. Recently, state-of-the-art query generation methods leverage Large Language Models (LLMs) for their strong capabilities in context understanding and text generation. However, they still face challenges in generating high-quality queries in terms of inferring user intent based on their web search interaction history. In this paper, we propose Token-level Proximal Policy Optimization (TPPO), a noval approach designed to empower LLMs perform better in query generation through fine-tuning. TPPO is based on the Reinforcement Learning from AI Feedback (RLAIF) paradigm, consisting of a token-level reward model and a token-level proximal policy optimization module to address the sparse reward challenge in traditional RLAIF frameworks. To evaluate the effectiveness and robustness of TPPO, we conducted experiments on both open-source dataset and an industrial dataset that was collected from a globally-used search engine. The experimental results demonstrate that TPPO significantly improves the performance of query generation for LLMs and outperforms its existing competitors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。