解决冗余数据中样本排序不稳定问题,提升排名可靠性。
SCARV: Structure-Constrained Aggregation for Stable Sample Ranking in Redundant NLP Datasets

- 基于多随机种子聚合与冗余聚类结构感知,增强排序稳定性。
- 在多个任务和数据集上显著提升全局与局部稳定性,改善子集选择效果。
- 适合数据清洗、异常检测等需要可复现排序的场景。
样本级排序在以数据为中心的NLP中被广泛用于分析、过滤、调试和数据整理,但现有方法通常对每个样本独立评分并排序,忽视了真实数据集中常见的完全重复、近似重复、改写句等冗余结构。这种假设在存在冗余时极不稳定,相同样本在不同随机种子下可能获得不一致的相对顺序。本文提出SCARV框架,通过在已有评分代理(proxy)之上进行模块化聚合,在多种子鲁棒聚合基础上引入结构感知的冗余聚类分配机制。在合成冗余、自然挖掘的QQP冗余数据、多种代理模型、多个NLP任务及端到端DistilBERT微调实验中,SCARV显著优于仅依赖种子的原始排序,在全局与局部稳定性上均有提升,并使基于排序的决策(如子集选择、可疑样本检索)更具可复现性。分析表明,多种子鲁棒聚合是主要稳定因素,而结构感知组件在聚合预算低或冗余聚类具有信息量且覆盖充分时才额外增益。因此,SCARV并非通用数据筛选器,而是面向冗余数据集中代理生成排序的稳定性增强层。
原文摘要 · Abstract (English)
Sample-level rankings are increasingly used in data-centric NLP for analysis, filtering, debugging, and curation, yet existing pipelines typically score training examples pointwise and rank them as if they were independent. This assumption is fragile in the presence of exact duplicates, near-duplicates, paraphrases, and other redundant structure common in NLP corpora, where stochastic training can make highly similar examples receive unstable relative orderings across random seeds. We study stable sample-level ranking under redundancy and propose \textsc{SCARV}, a modular aggregation framework that operates on top of an existing scoring proxy. \textsc{SCARV} combines robust multi-seed aggregation with a structure-aware aggregation/allocation step over redundancy clusters. Across synthetic redundancy, naturally mined QQP redundancy, multiple proxy families, several NLP tasks, and end-to-end DistilBERT fine-tuning, \textsc{SCARV} substantially improves over bare proxy rankings in global and local stability and yields more reproducible ranking-based decisions such as subset selection and suspicious-example retrieval. Our decomposition and compute-aware frontier sharpen the mechanism: robust multi-seed aggregation is the dominant generic stabilizer, while the structure-aware component adds value mainly under low aggregation budgets or when redundancy clusters are informative, naturally occurring, or sufficiently covered. These results position \textsc{SCARV} not as a universal data selector or a universally dominant replacement for seed-only aggregation, but as a stability-oriented aggregation layer for proxy-induced rankings in redundant NLP datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。