无需历史数据,自动压缩评估集,成本降90%仍保精度
Less is more: Not all samples are effective for evaluation
- 用少量领域数据微调模型,生成语义嵌入并聚类
- 在3GPP通信数据集上压缩超90%样本,精度几乎不变
- 适合新任务、新模型的冷启动评估场景
大语言模型在垂直领域的应用催生了大量专用评估基准,但这些基准普遍存在语义冗余且评估开销高。现有压缩方法(如tinyBenchmarks)依赖多个历史模型的正确性标签,无法用于无先验评估数据的新任务或新模型场景。为此,我们提出一种无需历史数据的测试集压缩框架。首先在少量领域数据上微调基础大模型以内化任务语义;随后仅基于原始文本生成所有测试样本的高层语义嵌入;在领域适配的嵌入空间中进行任务感知聚类,并引入新颖的“数据集透视”机制,通过分析聚类几何结构动态调节压缩强度。在专业领域数据集(特别是大规模3GPP通信基准)上的实验表明,该方法能有效识别并移除冗余样本,评估成本降低超过90%,同时保持对完整基准的高度保真度。
原文摘要 · Abstract (English)
The versatility of Large Language Models (LLMs) in vertical domains has spurred the development of numerous specialized evaluation benchmarks. However, these benchmarks often suffer from significant semantic redundancy and impose high computational costs during evaluation. Existing compression methods, such as tinyBenchmarks depend critically on correctness labels from multiple historical models evaluated on the full test set, making them inapplicable in cold-start scenarios, such as the introduction of a new task, domain, or model with no prior evaluation history. To address this limitation, we propose a history-free test set compression framework that requires no prior model performance data. Our method begins by fine-tuning a base LLM on a small amount of domain-specific data to internalize task-relevant semantics. It then generates high-level semantic embeddings for all original test samples using only their raw textual content. In this domain-adapted embedding space, we perform task-aware clustering and introduce a novel dataset X-ray mechanism that analyzes cluster geometry to dynamically calibrate the compression intensity based on the intrinsic redundancy of the benchmark. Experiments on professional-domain dataset, notably a large-scale 3GPP communications benchmark, demonstrate that our approach effectively identifies and removes redundant samples, reducing evaluation cost by over 90% while preserving high fidelity to the full benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。