针对点击率预测的模型失效问题,提出结构化表达新架构。
From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction
- 用字段感知参数重构Transformer,实现与数据语义匹配的结构表达
- 实测在生产环境提升4.38% AUC,点击率和每千次展现收益分别增2.33%、0.66%
- 适合推荐系统研究者及工业界部署优化需求者
尽管大规模投入模型规模,点击率(CTR)预测的深度模型仍面临回报递减,与大语言模型中可预测的缩放规律形成鲜明对比。我们发现根源在于结构错配:标准Transformer假设序列组合性,而CTR数据需要对异构字段进行组合推理。为此,提出字段感知Transformer(FAT),通过以字段为中心重构Transformer模块,实现结构化表达,从根本上将模型复杂度依赖从总词表大小 $n$ 转向字段数量 $F$($n \gg F$)。为解耦模型容量与字段基数,FAT采用基函数组成的超网络,从共享基底合成字段特定参数,进一步降低参数复杂度。理论上,基于Rademacher复杂度建立缩放律;实证上,FAT优于现有最优方法,达+4.38% AUC提升,并在真实线上环境中实现+2.33% CTR和+0.66% RPM增长。本工作表明,可扩展推荐并非仅靠规模,而是源于与数据语义一致的结构表达。
原文摘要 · Abstract (English)
Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the {predictable scaling laws} seen in large language models (LLMs). We identify the root cause as a {fundamental} \textit{structural misalignment}: {standard} Transformers assume sequential compositionality, whereas CTR data demand combinatorial reasoning over {heterogeneous} fields. To restore alignment, we introduce the \textbf{Field-Aware Transformer (FAT)}. {By reconstructing the standard Transformer block with field-centric parameters, FAT achieves \textit{structured expressivity}, {fundamentally shifting the model complexity dependence from the total vocabulary size $n$ with the number of fields $F$ ($n \gg F$).}} Crucially, to decouple model capacity from field cardinality, FAT employs a {Basis-Composed Hypernetwork} to synthesize field-specific parameters from shared bases, further reducing parameter complexity. {Theoretically, we ground this scaling behavior through a formal scaling law based on Rademacher complexity. Empirically, FAT outperforms exisiting state-of-the-art methods with up to \textbf{+4.38\%} AUC improvement, and delivers \textbf{+2.33\%} CTR and \textbf{+0.66\%} RPM in live production.} Our work establishes that scalable recommendation arises not from size alone, but from \textit{structured expressivity} -- architectural coherence with data semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。