数据泄露让代码密钥检测模型虚高，真实效果被严重夸大。
From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models
- 分析公开数据集中的重复样本，发现训练与测试集存在重叠
- 实验显示泄漏导致检测准确率虚高，真实场景性能大打折扣
- 提醒开发者警惕评估结果水分，适合安全模型验证者阅读
机器学习模型在软件安全任务中日益普及。这些模型通常在大规模互联网获取的数据集上训练和评估，而这些数据集常包含重复或高度相似的样本。当此类样本被分配到训练集和测试集中时，可能导致数据泄露，使模型记忆模式而非学会泛化。本文研究了一个广泛使用的硬编码密钥基准数据集中的重复问题，揭示了数据泄露如何显著夸大基于AI的密钥检测模型的报告性能，从而造成其实际应用效能的误导性认知。
原文摘要 · Abstract (English)
Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。