用大模型生成假反馈,让冷启动的推荐系统更快变聪明。
Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits

- 大模型预测未选选项的收益,以伪数据补充学习
- 通过校准门控动态调节伪数据权重,早期更依赖可信预测
- 任务定制提示能降19%累计损失,提示设计比参数更重要
上下文老虎机算法在冷启动阶段因数据不足导致高累积后悔。本文提出将大语言模型(LLM)伪观测引入独立线性UCB(Disjoint LinUCB):每轮结束后,LLM预测未选择动作的反事实奖励,并以加权伪观测形式注入学习器。权重由校准门控衰减策略控制,该策略通过指数移动平均追踪LLM在已选动作上的预测精度;校准误差高时抑制LLM影响,预测准确则在关键早期阶段赋予更高权重。在两个环境上评估:UCI Mushroom(2臂,非对称奖励)和MIND-small(5臂新闻推荐),结果表明,使用任务特定提示时,相较纯LinUCB,MIND上累积后悔降低19%。然而,通用反事实提示框架反而增加后悔,表明提示设计是主导因素,甚至超过衰减策略或校准门控参数的选择。我们分析了校准门控在小预测误差场景下的失效模式,并提供了伪观测权重所遵循偏差-方差权衡的理论动机。
原文摘要 · Abstract (English)
Contextual bandit algorithms suffer from high regret during cold-start, when the learner has insufficient data to distinguish good arms from bad. We propose augmenting Disjoint LinUCB with LLM pseudo-observations: after each round, a large language model predicts counterfactual rewards for the unplayed arms, and these predictions are injected into the learner as weighted pseudo-observations. The injection weight is controlled by a calibration-gated decay schedule that tracks the LLM's prediction accuracy on played arms via an exponential moving average; high calibration error suppresses the LLM's influence, while accurate predictions receive higher weight during the critical early rounds. We evaluate on two contextual bandit environments - UCI Mushroom (2-arm, asymmetric rewards) and MIND-small (5-arm news recommendation) - and find that when equipped with a task-specific prompt, LLM pseudo-observations reduce cumulative regret by 19% on MIND relative to pure LinUCB. However, generic counterfactual prompt framing increases regret on both environments, demonstrating that prompt design is the dominant factor, more important than the choice of decay schedule or calibration gating parameters. We analyze the failure modes of calibration gating on domains with small prediction errors and provide a theoretical motivation for the bias-variance trade-off governing pseudo-observation weight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。