主动发现大模型的系统性错误片段，用极少标注实现高效定位。
Active Slice Discovery in Large Language Models
- 通过主动学习策略自动聚类可能属于同一错误片段的样本。
- 仅用2%-10%的标注数据即达到接近全量标注的准确率。
- 适合关注模型公平性与缺陷诊断的研究者和工程师。
大语言模型在特定数据子集上常表现出系统性错误，称为错误片段（error slices）。例如，模型对某个人群相关的有毒评论识别能力较差。识别这些错误片段对理解与改进模型至关重要，但挑战较大。本文提出主动片段发现（Active Slice Discovery）方法，旨在减少人工标注需求：通过有限的标注资源，主动选择可能属于同一错误片段的样本进行验证。我们在毒性分类任务中实证评估了该方法，考察不同特征表示与主动学习算法的效果。结果表明，在多个错误片段上，基于不确定性的主动学习算法表现最佳，仅使用2%-10%的可用片段成员信息即可取得竞争性精度，显著优于基线方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often exhibit systematic errors on specific subsets of data, known as error slices. For instance, a slice can correspond to a certain demographic, where a model does poorly in identifying toxic comments regarding that demographic. Identifying error slices is crucial to understanding and improving models, but it is also challenging. An appealing approach to reduce the amount of manual annotation required is to actively group errors that are likely to belong to the same slice, while using limited access to an annotator to verify whether the chosen samples share the same pattern of model mistake. In this paper, we formalize this approach as Active Slice Discovery and explore it empirically on a problem of discovering human-defined slices in toxicity classification. We examine the efficacy of active slice discovery under different choices of feature representations and active learning algorithms. On several slices, we find that uncertainty-based active learning algorithms are most effective, achieving competitive accuracy using 2-10% of the available slice membership information, while significantly outperforming baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。