用数据库内查询检测类别分布变化，无需训练模型。
RankShift: In-Database Detection and Explanation of Categorical Shifts
- 基于皮尔逊相关性分析类别占比偏移，定位异常来源。
- 在雷电事件中检出率0.787 AUROC，优于自编码器的0.771。
- 适合日志监控场景，零训练开销，适合实时部署。
登录服务可能维持失败登录总数不变，但某一来源占比从2%升至30%；系统日志中罕见事件模板变频繁，而消息速率稳定。这类变化仅改变活跃类别分布，不改变事件总数。RankShift在存储数据的分析型数据库内检测此类变化，通过皮尔逊相关性评分比较各窗口的类别占比与正常基准，其各项指标可识别导致变化的具体类别。同一查询返回得分、校准警报及最大增长贡献项。在HDFS、BGL和Thunderbird数据集上评估显示，其在HDFS上的AUROC达0.999（自编码器为1.000），在Thunderbird上为0.983（自编码器0.949）。在固定总量的控制实验中，对罕见类别变化的检出率可达0.787 AUROC，高于自编码器的0.771。三组数据集中，误报率均符合设定操作水平。RankShift无需模型训练或推理服务，部署状态仅为自编码器的1/137。
原文摘要 · Abstract (English)
A login service can receive its usual number of failed sign-ins while one source grows from 2% to 30% of them. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable. These events change which categories are active without changing how many events occur. RankShift detects such changes inside the analytical database that stores the data. It compares each window's category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change. The same query returns the score, calibrated alert, and largest increasing contributions. We evaluate RankShift on HDFS, BGL, and Thunderbird. It matches the count-vector autoencoder within 0.001 AUROC on HDFS (0.999 versus 1.000) and leads on Thunderbird (0.983 versus 0.949). In a controlled fixed-volume experiment, RankShift detects rare-category shifts that are invisible to event-count monitoring, reaching 0.787 AUROC compared with 0.771 for the autoencoder. Across all three corpora, observed false-alarm rates track the requested operating levels. RankShift requires no model training or inference service, and the autoencoders deployed state is 137x larger.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。