用AI精准识别文件类型，1MB内存就能跑。
Magika: AI-Powered Content-Type Detection
- 基于深度学习模型，单核CPU+1MB内存即可运行
- 100多种文件类型，100万+文件测试下平均F1达99%
- 已用于Gmail和VirusTotal，支持超200种类型
内容类型检测——即识别任意字节序列中编码的数据类型——对操作系统、开发环境、逆向工程及各类安全应用至关重要。本文介绍Magika，一种新型AI驱动的内容类型检测工具。其核心为一个可在单个CPU上运行、仅需1MB内存存储模型权重的深度学习模型。实验表明，Magika在超过100种内容类型和百万级文件测试集上实现了平均99%的F1分数，优于现有所有工具。为促进采用与改进，Magika以Apache 2许可证开源，模型与训练流程均公开。该工具已被Gmail用于附件扫描，并集成至VirusTotal辅助恶意软件分析。本文介绍的是Magika的首个版本，最新版本已支持超过200种内容类型，详情可访问GitHub仓库：https://github.com/google/magika。
原文摘要 · Abstract (English)
The task of content-type detection -- which entails identifying the data encoded in an arbitrary byte sequence -- is critical for operating systems, development, reverse engineering environments, and a variety of security applications. In this paper, we introduce Magika, a novel AI-powered content-type detection tool. Under the hood, Magika employs a deep learning model that can execute on a single CPU with just 1MB of memory to store the model's weights. We show that Magika achieves an average F1 score of 99% across over a hundred content types and a test set of more than 1M files, outperforming all existing content-type detection tools today. In order to foster adoption and improvements, we open source Magika under an Apache 2 license on GitHub and make our model and training pipeline publicly available. Our tool has already seen adoption by the Gmail email provider for attachment scanning, and it has been integrated with VirusTotal to aid with malware analysis. We note that this paper discusses the first iteration of Magika, and a more recent version already supports more than 200 content types. The interested reader can see the latest development on the Magika GitHub repository, available at https://github.com/google/magika.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。