用相似性搜索优化异常检测边界,大幅减少标注数据需求。
Refining Decision Boundaries In Anomaly Detection Using Similarity Search Within the Feature Space
- 构建稀疏双对抗自编码器学习紧凑特征表示
- 相似性引导主动学习使nDCG达1.0,标签量减少80%
- 适合网络安全中罕见攻击的高效检测场景
在高度不平衡的数据集(如网络安全中的高级持续性威胁)中检测稀有且多样的异常仍是机器学习的核心挑战。主动学习通过智能查询标记源以降低标注成本,但传统方法常忽视特征空间的内在几何结构。本文提出SDA2E——一种基于稀疏双对抗注意力的自编码器,用于从高维不平衡数据中学习紧凑且可区分的潜在表示。进一步设计了一种相似性引导的主动学习框架,集成三种新策略:正常样本扩展(通过相似正常点增强训练集以提升重构保真度)、异常优先排序(聚焦于类似已知异常的样本以提高排名精度)及混合策略(平衡模型优化与排名)。核心创新是为稀疏二值嵌入设计的新相似性度量SIM_NM1。在52个不平衡数据集(包括多个DARPA Transparent Computing场景)上评估,对比15种先进方法,结果表明SDA2E在多个案例中实现nDCG高达1.0,同时相比被动训练将所需标注数据减少最高80%。统计检验验证了改进的显著性。本工作建立了一个稳健、高效且经统计验证的异常检测框架,尤其适用于高级持续性威胁等网络安全应用。
原文摘要 · Abstract (English)
Detecting rare and diverse anomalies in highly imbalanced datasets-such as Advanced Persistent Threats (APTs) in cybersecurity-remains a fundamental challenge for machine learning systems. Active learning offers a promising direction by strategically querying an oracle to minimize labeling effort, yet conventional approaches often fail to exploit the intrinsic geometric structure of the feature space for model refinement. In this paper, we introduce SDA2E, a Sparse Dual Adversarial Attention-based AutoEncoder designed to learn compact and discriminative latent representations from imbalanced, high-dimensional data. We further propose a similarity-guided active learning framework that integrates three novel strategies to refine decision boundaries efficiently: mormal-like expansion, which enriches the training set with points similar to labeled normals to improve reconstruction fidelity; anomaly-like prioritization, which boosts ranking accuracy by focusing on points resembling known anomalies; and a hybrid strategy that combines both for balanced model refinement and ranking. A key component of our framework is a new similarity measure, Normalized Matching 1s (SIM_NM1), tailored for sparse binary embeddings. We evaluate SDA2E extensively across 52 imbalanced datasets, including multiple DARPA Transparent Computing scenarios, and benchmark it against 15 state-of-the-art anomaly detection methods. Results demonstrate that SDA2E consistently achieves superior ranking performance (nDCG up to 1.0 in several cases) while reducing the required labeled data by up to 80% compared to passive training. Statistical tests confirm the significance of these improvements. Our work establishes a robust, efficient, and statistically validated framework for anomaly detection that is particularly suited to cybersecurity applications such as APT detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。