用全局校准替代逐实验标注,实现低成本高频率的模型内容占比监测。
Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments
- 建立全局校准的机器学习评分桶,持续从LLM标注流中更新。
- 同一套校准结果复用于上千次实验,日均覆盖250个实验组。
- 可发现单次标注无法捕捉的1%~5%级内容变化,适合大规模实验团队。
在线媒体平台需追踪内容属性的曝光占比(即流行度),以评估A/B实验中的权衡并设定防护机制。基于大模型的标注虽精准,但因成本过高,难以在每日数百个并发实验中按实验、分组、时段执行。本文介绍在Pinterest实验平台部署的代理式流行度测量系统:维护一个由每日批量样本持续刷新的全局校准模型评分桶,通过每实验的SQL指标和聚焦差异的仪表盘,将桶级别流行度复用于所有实验。该校准源自平台每日批次数据,能随流量分布漂移保持代表性,在实际部署中实现零额外标注成本。无基础设施团队也可通过周期性校准流程,将成本摊薄至所有下游实验,而非每次实验重复支付。系统日均服务约100个实验、250个实验组,覆盖六个已校准的内容类别(含保留测试)。相比传统每实验一次的LLM标注(仅覆盖少量实验的局部分组),该方案在相同标注预算下实现超过20倍的并发实验组覆盖。在约300次生产审计中,92%的评估显示代理结果的95%置信区间包含基于LLM的参考值;日粒度的差值聚合可检测出单次标注无法发现的2%~5%相对变化。
原文摘要 · Abstract (English)
Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20$\times$ as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95\% confidence interval contains the LLM-based reference point estimate in 92\% of evaluations, and day-level delta aggregation recovers 2--5\% relative shifts that no single per-arm LLM measurement can detect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。