提出新方法DCScore，高效评估大模型生成数据集的多样性。
Measuring Diversity in Synthetic Datasets
- 将数据多样性评估转化为样本分类任务，利用样本间关系
- 在多个数据集上与多样性基准相关性更强，效果更优
- 计算成本显著低于现有方法，适合大规模评估
大语言模型广泛用于生成自然语言处理任务的合成数据集，如文本分类和摘要生成。然而，准确衡量这些合成数据集的多样性——对模型鲁棒性至关重要的方面——仍是重大挑战。本文提出DCScore，一种从分类视角衡量合成数据集多样性的新方法。DCScore将多样性评估建模为样本分类任务，利用样本间的相互关系。我们进一步提供了理论验证，证明DCScore满足多样性相关公理，凸显其作为原则性评估方法的价值。在合成数据集上的实验表明，DCScore与多种多样性伪真值具有更强的相关性，验证了其有效性。此外，实证与理论证据均表明，相较于现有方法，DCScore显著降低了计算开销。代码已公开：https://github.com/bluewhalelab/dcscore。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。