用搜索优化生成式出价,让系统更灵活适应不同广告主偏好。
GAS: Generative Auto-bidding with Post-training Search
- 出价模型输出后通过搜索精炼,结合小批评家提升对齐效果。
- 线上测试显示目标成本提升4.60%,长序列决策表现更优。
- 适合需要快速适配多种广告主偏好的实时竞价场景。
自动出价在在线广告中至关重要,能代表广告主自动出价。生成式自动出价利用Transformer、扩散模型等生成基于可调条件的出价,有望直接从数据中学习最优策略并灵活调整偏好。然而,生成模型常因低质量数据导致条件(如预期回报)与真实动作价值不匹配,尤其在长序列决策中更为明显;此外,数据集中多数偏好会抑制模型对少数广告主偏好的泛化能力。尽管可通过收集高质量数据并重新训练多个模型来解决,但成本高昂,难以推动自动出价进入大模型时代。为此,我们提出一种灵活实用的生成式自动出价方案GAS,采用后训练搜索机制,在不重训模型的前提下精炼基础策略输出,以适应多样偏好。通过弱到强的搜索对齐方式,训练针对不同偏好的小型批评家,并结合类MCTS的搜索策略优化输出。特别地,引入基于Transformer的批评家与策略指示联合训练的投票机制,显著提升搜索对齐性能。此外,为兼顾高频率偏好场景的计算效率,我们设计了一种高效微调方法。在真实世界数据集及快手广告平台的线上A/B测试中,GAS表现优异,目标成本提升达4.60%。
原文摘要 · Abstract (English)
Auto-bidding is essential in facilitating online advertising by automatically placing bids on behalf of advertisers. Generative auto-bidding, which generates bids based on an adjustable condition using models like transformers and diffusers, has recently emerged as a new trend due to its potential to learn optimal strategies directly from data and adjust flexibly to preferences. However, generative models suffer from low-quality data leading to a mismatch between the condition, like return to go, and true action value, especially in long sequential decision-making. Besides, the majority preference in the dataset may hinder models' generalization ability on minority advertisers' preferences. While it is possible to collect high-quality data and retrain multiple models for different preferences, the high cost makes it unaffordable, hindering the advancement of auto-bidding into the era of large foundation models. To address this, we propose a flexible and practical Generative Auto-bidding scheme using post-training Search, termed GAS, to refine a base policy model's output and adapt to various preferences. We use weak-to-strong search alignment by training small critics for different preferences and an MCTS-inspired search to refine the model's output. Specifically, a novel voting mechanism with transformer-based critics trained with policy indications could enhance search alignment performance. Additionally, utilizing the search, we provide a fine-tuning method for high-frequency preference scenarios considering computational efficiency. Extensive experiments conducted on the real-world dataset and online A/B test on the Kuaishou advertising platform demonstrate the effectiveness of GAS, achieving significant improvements, e.g., 4.60% increment of target cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。