arXiv:2412.03611cs.LGcs.DB2024-12中稿 · as a regular paper…被引 1

无需真实标签,实时高效估算数据流中项目频率。

Learning-based Sketches for Frequency Estimation in Data Streams without Ground Truth

  • 基于等效学习的在线训练,无需真实频率数据
  • 内存极紧时逼近理想观测器性能,误差显著更低
  • 速度比传统方法快近500倍,适合实时系统

在高吞吐、快速的数据流中估计项目频率是数据库和网络测量等领域的关键问题。传统哈希表仅能提供粗略估计,而现有学习增强方法通常依赖离线训练所需的真值(GT)或标签,这些信息常不可得。此外,它们更新速度慢,难以满足实时处理需求。为此,我们提出UCL-sketch,一种实用的学习型频率估算框架。其核心创新包括:(i) 无需真值的在线学习机制,基于等效学习;(ii) 利用逻辑结构化估计桶的可扩展架构,支持真实世界数据流。UCL-sketch采用压缩感知(CS),其收敛后的估计器具有理论保证的误差界,远优于以往方法,且不牺牲处理速度。在真实与合成数据集上的大量实验表明,该方法在单键精度和分布建模方面均优于现有方法。尤其在极端内存约束下,其性能几乎媲美不可行的理想观测器(omniscient oracle)。相比现有基于方程的哈希表,平均解码速度提升近500倍。代码已开源:https://github.com/Y-debug-sys/UCL-sketch。

原文摘要 · Abstract (English)

Estimating the frequency of items on the high-volume, fast data stream has been extensively studied in many areas, such as database and network measurement. Traditional sketches provide only coarse estimates under strict memory constraints. Although some learning-augmented methods have emerged recently, they typically rely on offline training with real frequencies or/and labels, which are often unavailable. Moreover, these methods suffer from slow update speeds, limiting their suitability for real-time processing despite offering only marginal accuracy improvements. To overcome these challenges, we propose UCL-sketch, a practical learning-based paradigm for per-key frequency estimation. Our design introduces two key innovations: (i) an online training mechanism based on equivalent learning that requires no ground truth (GT), and (ii) a highly scalable architecture leveraging logically structured estimation buckets to scale to real-world data stream. The UCL-sketch, which utilizes compressive sensing (CS), converges to an estimator that provably yields a error bound far lower than that of prior works, without sacrificing the speed of processing. Extensive experiments on both real-world and synthetic datasets demonstrate that our approach outperforms previously proposed approaches regarding per-key accuracy and distribution. Notably, under extremely tight memory budgets, its quality almost matches that of an (infeasible) omniscient oracle. Moreover, compared to the existing equation-based sketch, UCL-sketch achieves an average decoding speedup of nearly 500 times. To help further research and development, our code is publicly available at https://github.com/Y-debug-sys/UCL-sketch.

频率估计数据流学习型哈希压缩感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。