arXiv:2508.17125cs.IR2025-08被引 6

VQL通过分层量化注意力,高效建模超长用户行为序列。

VQL: An End-to-End Context-Aware Vector Quantization Attention for Ultra-Long User Behavior Modeling

  • 仅量化注意力键值,保持值不变,实现无长度依赖推理。
  • 多尺度分组量化降低误差,固定缓存大小下提升精度。
  • 不扩编码本直接注入静态特征与相对位置信息,适合工业部署。

在大规模推荐系统中,超长用户行为序列蕴含丰富的兴趣演化信号。延长序列通常能提升精度,但生产环境因延迟和内存限制难以直接建模。现有方法分为两类:(1) top-k 检索,截断序列可能丢失大部分注意力质量;(2) 编码器压缩,虽保留覆盖度但过度压缩,忽略时间间隔等关键上下文。两者均难以兼顾低损失压缩、上下文感知与效率。本文提出 VQL,一种面向超长行为建模的上下文感知向量量化注意力框架,包含三项创新:(1) 仅量化注意力键,值保持完整;证明 softmax 归一化误差界与序列长度无关,并通过码本损失直接监督量化质量,支持离线缓存实现无长度依赖推理;(2) 多尺度量化:将注意力头分组,每组使用独立小码本,在固定缓存下降低量化误差;(3) 高效上下文注入:静态特征(如物品类别、模态)直接融合,相对位置通过可分离的时间核建模,所有上下文注入无需扩大码本,缓存表示保持查询无关。在三个大规模数据集(KuaiRand-1K、KuaiRec、TMALL)上的实验表明,VQL持续优于强基线,精度更高且推理延迟更低,建立了超长序列推荐中精度与效率平衡的新基准。

原文摘要 · Abstract (English)

In large-scale recommender systems, ultra-long user behavior sequences encode rich signals of evolving interests. Extending sequence length generally improves accuracy, but directly modeling such sequences in production is infeasible due to latency and memory constraints. Existing solutions fall into two categories: (1) top-k retrieval, which truncates the sequence and may discard most attention mass when L >> k; and (2) encoder-based compression, which preserves coverage but often over-compresses and fails to incorporate key context such as temporal gaps or target-aware signals. Neither class achieves a good balance of low-loss compression, context awareness, and efficiency. We propose VQL, a context-aware Vector Quantization Attention framework for ultra-long behavior modeling, with three innovations. (1) Key-only quantization: only attention keys are quantized, while values remain intact; we prove that softmax normalization yields an error bound independent of sequence length, and a codebook loss directly supervises quantization quality. This also enables L-free inference via offline caches. (2) Multi-scale quantization: attention heads are partitioned into groups, each with its own small codebook, which reduces quantization error while keeping cache size fixed. (3) Efficient context injection: static features (e.g., item category, modality) are directly integrated, and relative position is modeled via a separable temporal kernel. All context is injected without enlarging the codebook, so cached representations remain query-independent. Experiments on three large-scale datasets (KuaiRand-1K, KuaiRec, TMALL) show that VQL consistently outperforms strong baselines, achieving higher accuracy while reducing inference latency, establishing a new state of the art in balancing accuracy and efficiency for ultra-long sequence recommendation.

推荐系统向量量化长序列建模高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。