用词频分布加速恶意软件特征提取，提升检测精度与效率
Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora
- 基于字节n-gram的压缩特征分类器，兼顾速度与内存占用
- 新算法使顶k词频提取提速35倍，训练集扩大后AUC提升30%
- 适合需要实时检测、频繁更新模型的工业级安全系统
使用字节n-gram作为特征的分类器是目前唯一能满足部署需求的方法：模型大小小于2 MB，处理速度达每秒数GB，延迟低于10毫秒。尽管6-8阶n-gram在生产环境中表现最佳，但因需在数TB可执行文件中频繁计算最常见n-gram，导致模型更新成本过高。由于n-gram分布符合齐普夫分布，我们利用其特性设计了新的顶k n-gram提取算法，相较之前最优方案快达35倍。通过该方法，我们成功扩展了生产训练集，使新恶意软件检测的AUC最高提升30%。理论与实证均表明，该方法能以极低误差选出前k个高频项，其成果源于理论与工程的紧密协作。
原文摘要 · Abstract (English)
A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to $35\times$ faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30\% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。