arXiv:2606.05257cs.LGcs.IR2026-06

研究用户行为模型的计算规律,发现小嵌入器更高效,评估指标影响最佳训练方案。

Scaling Laws for Behavioral Foundation Models over User Event Sequences

论文配图:Scaling Laws for Behavioral Foundation Models over User Event Sequences
图 1 · 摘自论文原文
  • 采用双阶段架构:嵌入器+解码器,系统测试不同参数分配与采样策略。
  • 嵌入器仅占2%参数却最优,因其每步计算成本高且重复处理更多数据。
  • 评估指标改变时,最优训练配置随之变化,说明评价标准也是缩放规律的一部分。

基础模型越来越多地在推荐、支付、反欺诈和电商等场景的用户行为序列上训练,但缺乏像语言模型那样的计算量校准规律。本文研究一种常见的两阶段行为模型架构:基于特征的事件嵌入器将多模态项目映射为向量,解码器仅使用Transformer预测下一个事件。在真实交互数据上进行约600次实验,覆盖10^15至10^19训练浮点运算次数(FLOPs),系统调整四个部署相关维度:两部分参数比例、关键批次大小、模型与数据分配,以及冻结嵌入器后采样的负样本数量。结果表明,在所有测试预算下,小嵌入器(占比约2%)在计算效率上最优,因其每步成本更高且面对的重复项目远多于上下文编码器。低算力时训练偏重数据,但随着算力增加,数据与负样本比(D/N)逐渐趋近Chinchilla启发式。训练目标与部署排序指标之间存在随规模变化的不一致:关键批次大小、冻结后的最优负样本数及损失与排序质量的一致性均随算力和评估指标变化。在大预算下,负样本数量需求上升;当达到10^19 FLOPs时,瓶颈从算力转为候选集内存。因此,在行为基础模型中,评估指标本身是缩放规律的一部分——改变它会改变最优训练配方。

原文摘要 · Abstract (English)

Foundation models are increasingly trained on sequences of user actions in recommendation, payments, fraud, and commerce, but these models still lack the kind of compute calibration that scaling laws provide for language models. We study a common two-part behavioral-model architecture: a feature-based event embedder maps each multi-modal item to a vector, and a decoder-only transformer predicts the next event from the resulting sequence. Across roughly 600 runs on real interaction data, spanning $10^{15}$-$10^{19}$ training FLOPs, we jointly vary four deployment-relevant axes: the two-part parameter split, critical batch size, model/data allocation, and the number of sampled negatives used after freezing the embedder. A small embedder ($s^{\star}\!\approx\!2\%$ of parameters) is compute-optimal at every budget we test because embedder parameters are both more expensive per step and exposed to far more repeated items than contextualizer parameters. Compute-optimal training is data-heavy relative to text at low compute, but its $D/N$ ratio moves toward the Chinchilla heuristic as compute increases. The sampled training objective and deployed ranking metrics disagree in ways that themselves scale: critical batch size, optimal negative count after freezing, and the agreement between loss and ranking quality all shift with compute and with the chosen evaluation metric. For negative sampling, larger budgets increasingly prefer more negatives; by $10^{19}$ FLOPs the active constraint is candidate-axis memory rather than FLOPs. In behavioral foundation models, the evaluation metric is therefore part of the scaling law: changing it can change the compute-optimal recipe.

行为建模缩放定律推荐系统负采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。