arXiv:2506.17309cs.CRcs.LG2025-06中稿 · presentation at th…被引 3

用降维方法让恶意软件检测更快更准,不牺牲精度。

Efficient Malware Detection with Optimized Learning on High-Dimensional Features

  • 先用XGBoost选特征,再用PCA降维到384维
  • 轻量模型轻量训练61分钟,准确率达97.52%
  • 适合需要高效部署的反病毒系统

基于机器学习的恶意软件检测需从二进制文件中提取特征,因模型无法直接处理原始二进制。常用方法使用LIEF进行原始特征提取,并通过EMBER向量化器生成2381维特征向量。高维特征带来显著计算挑战。本文采用两种降维技术:基于XGBoost的特征选择和主成分分析(PCA),评估了128、256、384三种降维后的维度(分别为原维度的5.4%、10.8%、16.1%)。在XGBoost、LightGBM、Extra Trees、Random Forest四个模型上,使用统一划分的EMBER-2018、ERMDS和BODMAS数据集进行训练、验证与测试,确保泛化性并避免数据集偏差。实验表明,经XGBoost特征选择后,在384维特征上训练的LightGBM模型在统一数据集上达到最高准确率97.52%,实现计算效率与检测性能的最优平衡。该模型仅需61分钟训练,占用30 GB内存与19.5 GB磁盘空间,对完全未见数据集TRITIUM仍保持95.31%准确率,INFERNO上为93.98%。结果证明该方法具备可扩展性和计算高效性,且不损失检测精度。

原文摘要 · Abstract (English)

Malware detection using machine learning requires feature extraction from binary files, as models cannot process raw binaries directly. A common approach involves using LIEF for raw feature extraction and the EMBER vectorizer to generate 2381-dimensional feature vectors. However, the high dimensionality of these features introduces significant computational challenges. This study addresses these challenges by applying two dimensionality reduction techniques: XGBoost-based feature selection and Principal Component Analysis (PCA). We evaluate three reduced feature dimensions (128, 256, and 384), which correspond to approximately 5.4%, 10.8%, and 16.1% of the original 2381 features, across four models-XGBoost, LightGBM, Extra Trees, and Random Forest-using a unified training, validation, and testing split formed from the EMBER-2018, ERMDS, and BODMAS datasets. This approach ensures generalization and avoids dataset bias. Experimental results show that LightGBM trained on the 384-dimensional feature set after XGBoost feature selection achieves the highest accuracy of 97.52% on the unified dataset, providing an optimal balance between computational efficiency and detection performance. The best model, trained in 61 minutes using 30 GB of RAM and 19.5 GB of disk space, generalizes effectively to completely unseen datasets, maintaining 95.31% accuracy on TRITIUM and 93.98% accuracy on INFERNO. These findings present a scalable, compute-efficient approach for malware detection without compromising accuracy.

恶意软件检测降维XGBoostLightGBM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。