提出评估恶意软件分类器可信度的新框架,揭示现有模型在分布漂移下的可靠性危机。
On the Reliability and Stability of Selective Methods in Malware Classification Tasks
- 构建Aurora框架,通过验证置信度质量评估模型可靠性
- 发现顶尖模型在不同漂移数据集上置信度严重失准
- 适合关注实际部署中模型稳定性与可信度的研究者
现代漂移自适应恶意软件分类器的表现看似乐观,但这是否意味着真正的运行可靠性?当前评估范式主要关注基准性能指标,忽视了置信度与错误的对齐性以及运行稳定性。尽管先前工作强调了时间维度评估的重要性,并引入了选择性分类,本文则从互补角度探究恶意软件分类器在分布漂移下是否仍能保持可靠的置信度估计,并分析其科学进步与实际影响之间的张力。我们提出Aurora框架,基于置信度质量和操作韧性来评估分类器。Aurora对给定模型的置信度分布进行验证,以评估其估计的可靠性。不可靠的置信度会削弱运行信任,浪费主动学习中的标注预算,使误判样本在选择性分类中被遗漏。Aurora还配备一组超越单一时点性能的指标,旨在实现更全面的时间维度运行稳定性评估。我们在不同漂移强度的数据集上观察到最先进的框架存在脆弱性,提示可能需要重新审视底层假设。
原文摘要 · Abstract (English)
The performance figures of modern drift-adaptive malware classifiers appear promising, but does this translate to genuine operational reliability? The standard evaluation paradigm primarily focuses on baseline performance metrics, neglecting confidence-error alignment and operational stability. While prior works established the importance of temporal evaluation and introduced selective classification in malware classification tasks, we take a complementary direction by investigating whether malware classifiers maintain reliable and stable confidence estimates under distribution shifts and exploring the tensions between scientific advancement and practical impacts when they do not. We propose Aurora, a framework to evaluate malware classifiers based on their confidence quality and operational resilience. Aurora subjects the confidence profile of a given model to verification to assess the reliability of its estimates. Unreliable confidence estimates erode operational trust, waste valuable annotation budgets on non-informative samples for active learning, and leave error-prone instances undetected in selective classification. Aurora is further complemented by a set of metrics designed to go beyond point-in-time performance, striving towards a more holistic assessment of operational stability throughout temporal evaluation periods. The fragility we observe in SOTA frameworks across datasets of varying drift severity suggests it may be time to revisit the underlying assumptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。