用大模型自动找数据漏洞，帮工程师定位模型系统性错误。
What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing
- 用大模型自动标注数据，按语义生成切片，不依赖原有特征。
- 能低成本生成准确切片，识别出性能差的数据子集。
- 适合想排查模型深层缺陷的开发者和数据科学家。
机器学习模型会出错，但有时难以识别背后的系统性问题。从业者通过错误分析、测试、审计和红队攻击等手段提出假设，再用数据切片验证。传统数据切片受限于现有特征和编程切片函数。本文提出SemSlicer框架，支持语义数据切片，无需依赖已有特征即可识别语义一致的数据子集。该框架利用大语言模型对数据集进行标注，并根据用户定义的切片条件生成切片。实验表明，SemSlicer能以低开销生成高精度切片，灵活权衡设计维度，可靠识别性能不佳的数据子集，并帮助从业者发现反映系统性问题的有效切片。
原文摘要 · Abstract (English)
Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。