提出DeRes模型,解决推荐系统中残差连接的稳定性与适应性矛盾。
DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction

- 双路径设计:保留身份残差与跨层注意力残差并行
- 在多个数据集上提升0.32% AUC,计算开销仅增5%
- 适合追求高精度与低计算成本的工业级点击率预测
基于Transformer的点击率(CTR)模型在残差连接处面临瓶颈:预归一化下早期用户兴趣信号逐层稀释,身份跳跃无法遗忘过时兴趣,且每层仅关注前一层,丢失长程跨层依赖。现有注意力残差(AttnRes)虽缓解部分问题,但舍弃身份跳跃,未应用于推荐场景。受双路径网络(DPN)和残差的HORNN视角启发,本文提出DeRes,将每层分为两条并行路径——身份残差路径保持一阶特征复用与梯度流动,块注意力残差路径对所有先前块压缩输出进行跨层注意力以实现高阶回忆。向量级门控决定每维特征在两路径间的权重分配。进一步提出Pointwise AttnRes,用SiLU替代跨层注意力中的Softmax,使多层可同时激活,无关层获负权(遗忘),更契合CTR的并行多兴趣模式。在大规模工业数据集(331M交互)、Criteo(45M)和Avazu(40M)上,DeRes超越12个基线(包括OneTrans、TokenMixer-Large、UniMixer、mHC、AttnRes),AUC最高提升0.32%,额外浮点运算仅增加5%。在不同计算量下,DeRes呈现更陡峭的计算-AUC扩展律(gamma=0.118 vs. 0.071 for OneTrans,差距1.66倍),8层DeRes相当于16层OneTrans,等效AUC下节省约2倍计算量。消融实验证实双路径优于单路径,身份残差优于可学习残差,SiLU优于Softmax。
原文摘要 · Abstract (English)
Transformer-based CTR models face a growing bottleneck at the residual connection: under Pre-Norm, early user-interest signals are diluted layer by layer; the identity skip cannot forget stale interests; and each layer sees only its immediate predecessor, losing long-range cross-layer dependencies. Recent attention-based residual variants (AttnRes) address parts of this in language models, but drop the protective identity skip and have not been tried in recommendation. Drawing on Dual Path Networks (DPN) and the HORNN view of residuals, we present DeRes, which routes each layer through two parallel paths -- an Identity residual path that preserves first-order feature reuse and gradient flow, and a Block Attention Residual path that attends over compressed outputs of all earlier blocks for high-order recall. A vector-wise gate decides, per hidden dimension, the weight given to each path. We further propose Pointwise AttnRes, replacing the Softmax in the cross-layer attention with SiLU so that multiple past blocks can be activated simultaneously and irrelevant ones receive negative (forgetting) weights -- better aligned with CTR's parallel multi-interest patterns. On a large-scale industrial dataset (331M interactions from a major social-media platform), Criteo (45M), and Avazu (40M), DeRes outperforms twelve baselines including OneTrans, TokenMixer-Large, UniMixer, mHC, and AttnRes, achieving up to +0.32% AUC at under 5% extra FLOPs. Beyond a single operating point, DeRes fits a markedly steeper compute-AUC scaling law (gamma=0.118 vs. 0.071 for OneTrans, a 1.66x gap), so an 8-layer DeRes matches a 16-layer OneTrans -- about 2x compute saving at equivalent AUC. Ablations confirm that the dual-path design outperforms either single path, Identity beats learnable residuals, and SiLU beats Softmax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。