Edge-IIoTset的高精度检测结果实为序列化漏洞导致,非真实入侵识别。
Provenance, Not Behaviour: A Serialisation Artifact in Edge-IIoTset and a Leakage-Free Benchmark for Precision-Agriculture Intrusion Detection
- 用串行化痕迹中的字符串差异(0 vs 0.0)可无损区分攻击与正常流量
- 六种分类器在原数据上均达99.998%以上准确率,但修正后最高仅95.03%
- 重建的新基准AgriEdge消除泄漏,揭示真实泛化边界在感知执行层
Edge-IIoTset是工业物联网中机器学习入侵检测的参考基准,现有报告性能普遍超过99%。我们发现其中大部分性能并非源于真正的入侵检测能力。数据集预处理要求将七个类别列进行独热编码,其中四个列通过协议字段缺失值的字符串形式(正常流量为"0",攻击流量为"0.0")即可实现1.0000的准确率分离,该标签完全由序列化过程中的文件来源信息泄露导致,无需建模网络行为。在5折×3次重复交叉验证下,六种标准分类器均达到1.0000±0.0000准确率,第六个为0.99998。在修正协议后,朴素贝叶斯宏F1下降0.3005,最强模型稳定在0.9503±0.0011。标签、序数及频次编码均存在相同泄漏。由于原始子集缺失Modbus和设备身份信息,我们从原始捕获数据重新构建基准AgriEdge:共1,276,122行,五台设备全归属,且无任何列能以高于0.0288的区分度划分类别。留一设备外验证表明,泛化边界位于感知/执行层,随机森林平衡准确率从0.9988降至0.5083。非独立同分布联邦划分最多损失0.0037宏F1,但20轮LoRaWAN训练需4.6小时上行通信时间。
原文摘要 · Abstract (English)
Edge-IIoTset is the reference benchmark for machine-learning intrusion detection in the industrial Internet of Things, and results reported on it cluster above 99%. We show that much of that performance is not intrusion detection. The preprocessing recipe distributed with the dataset instructs researchers to one-hot encode seven categorical columns. Four of them separate attack from normal traffic with an accuracy of 1.0000 on their own, through the spelling of the placeholder written for an absent protocol field: the string "0" in the normal-traffic branch of the dataset build against "0.0" in the attack branch. The label is recoverable from a serialisation artifact encoding file provenance, with no network behaviour modelled, and separates every row of both curated subsets. Under 5-fold x 3-repeat cross-validation, five of six standard classifiers attain exactly 1.0000 +/- 0.0000 accuracy and the sixth attains 0.99998. Under a corrected protocol, naive Bayes falls by 0.3005 macro-F1 and the strongest model settles at 0.9503 +/- 0.0011. Label, ordinal and frequency encoding leak identically. Because the curated subsets also lack Modbus and per-device identity, we rebuild the benchmark from the raw captures under uniform parsing, producing AgriEdge: 1,276,122 rows, five devices with full attribution, and no column separating the classes above 0.0288. A leave-one-device-out sweep locates the generalisation boundary at the perception/actuation layer, where random forest falls from 0.9988 to 0.5083 balanced accuracy. Non-IID federated partitioning costs at most 0.0037 macro-F1, but a 20-round LoRaWAN training run costs 4.6 hours of uplink.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。