用分层大模型提升广告竞价智能性,兼顾推理与精准执行
LBM: Hierarchical Large Auto-Bidding Model via Reasoning and Acting
- 分两层设计:高层推理+低层执行,结合语言与数值输入
- 离线强化微调避免幻觉,训练效率高且泛化能力强
- 适合需要智能竞价策略的广告平台与算法工程师
在线广告平台广告拍卖规模不断增大,人工竞价已不现实,需依赖自动竞价实现经济目标。现有自动竞价方法多采用离线强化学习或生成式方法优化策略,但因黑箱训练和数据集模式覆盖有限,常出现反直觉行为,难以理解任务状态且在动态环境中泛化能力差。大型语言模型(LLMs)凭借人类先验知识和推理能力,为提升自动竞价性能提供了新路径。然而直接应用LLM于竞价面临挑战:拍卖中需精确动作,而通用模型缺乏专业竞价知识,易产生幻觉并导致次优决策。为此,我们提出分层大型自动竞价模型(LBM),利用LLM的推理能力构建更优竞价策略。包括高层的LBM-Think模型用于推理,低层的LBM-Act模型用于动作生成。具体地,提出双嵌入机制,高效融合语言与数值输入,支持语言引导的训练;并引入一种名为GQPO的离线强化微调技术,缓解LBM-Think的幻觉问题,提升决策性能,无需模拟或真实投放。实验表明,基于LBM的生成式主干模型在高效训练和泛化能力方面表现优异。
原文摘要 · Abstract (English)
The growing scale of ad auctions on online advertising platforms has intensified competition, making manual bidding impractical and necessitating auto-bidding to help advertisers achieve their economic goals. Current auto-bidding methods have evolved to use offline reinforcement learning or generative methods to optimize bidding strategies, but they can sometimes behave counterintuitively due to the black-box training manner and limited mode coverage of datasets, leading to challenges in understanding task status and generalization in dynamic ad environments. Large language models (LLMs) offer a promising solution by leveraging prior human knowledge and reasoning abilities to improve auto-bidding performance. However, directly applying LLMs to auto-bidding faces difficulties due to the need for precise actions in competitive auctions and the lack of specialized auto-bidding knowledge, which can lead to hallucinations and suboptimal decisions. To address these challenges, we propose a hierarchical Large autoBidding Model (LBM) to leverage the reasoning capabilities of LLMs for developing a superior auto-bidding strategy. This includes a high-level LBM-Think model for reasoning and a low-level LBM-Act model for action generation. Specifically, we propose a dual embedding mechanism to efficiently fuse two modalities, including language and numerical inputs, for language-guided training of the LBM-Act; then, we propose an offline reinforcement fine-tuning technique termed GQPO for mitigating the LLM-Think's hallucinations and enhancing decision-making performance without simulation or real-world rollout like previous multi-turn LLM-based methods. Experiments demonstrate the superiority of a generative backbone based on our LBM, especially in an efficient training manner and generalization ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。