用用户点击数据优化提示词，提升个性化推荐生成效果
Prompt Optimization with Logged Bandit Data
- 基于生成句相似性设计核函数，降低提示词空间梯度估计方差
- 在大规模候选提示词下，显著提升电影推荐描述的个性化水平
- 适合需要利用真实用户反馈优化LLM生成的工程场景
我们研究如何利用自然产生的用户反馈（如点击）来优化大型语言模型（LLM）流水线，以生成个性化的句子提示。朴素方法在提示空间中估计策略梯度时，要么因提示动作空间过大导致方差过高，要么因奖励预测不准导致偏差。为此，我们提出一种新型基于核函数的离策略梯度方法，通过利用生成句子间的相似性来估计策略梯度，显著降低方差同时抑制偏差。在新构建的基准测试套件上的实验表明，该方法在候选提示数量较大时，能有效生成个性化的电影推荐描述。
原文摘要 · Abstract (English)
We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。