arXiv:2608.18158cs.CLcs.AI2026-08

LLM在需背景知识的任务中优于传统方法,但有强关键词时优势不明显。

When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators

论文配图:When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators
图 1 · 摘自论文原文
  • 对比规则基线与LLM在电商数据清洗中的表现,测试零样本与少样本提示效果。
  • 实体匹配任务中规则基线F1达0.950,LLM零样本仅0.948,微调后反降至0.914。
  • 品牌错标检测中LLM因具备品牌知识优势,F1达0.833,远超规则基线的0.721。

本研究评估了LLM在电商数据质量标注中的表现,对比其与基于规则的基线和人工验证真值在实体匹配和品牌错标检测任务上的表现。在Abt-Buy基准(2,194对标签样本)上,规则基线(F1=0.950)与零样本提示的LLM(F1=0.948)性能相当;而基于小样本验证优化的少样本提示,在全量数据上反而使性能下降至F1=0.914,表明小样本评估可能具有误导性。在500条含合成错误的品牌错标数据上,由于可利用品牌间关系的背景知识,LLM显著优于简单规则基线(F1=0.833 vs 0.721)。重复运行测试(200对,5次,温度0.7)显示,模型一致性极高,平均99.7%的判断一致,99%样本在所有运行中结果相同。采用多数投票仅提升F1 0.005,代价却为5倍推理成本。结果表明,LLM是否有效取决于任务特性:当存在强词法信号时优势有限,但在依赖背景知识的任务中则具明显优势,且保持高度稳定。

原文摘要 · Abstract (English)

LLMs have been increasingly used to catch data quality issues automatically, but we know very little about how consistent these judgments actually are. This study tests an LLM on two e-commerce data quality tasks, entity matching and brand mislabeling, against rule based baselines and human verified ground truth, under both zero-shot and few-shot prompting. On entity matching while using the Abt Buy benchmark (2,194 labeled pairs), a simple rule based baseline (F1=0.950) performed about as well as LLM zero shot prompting (F1=0.948). Moreover, a few-shot prompt revision that looked effective on a small validation sample reduced full-scale performance to F1=0.914. This showed that small sample prompt evaluation can be misleading. On brand mislabeling detection, using 500 Amazon product listings with synthetically injected labeling errors, the LLM clearly outperformed a naive rule based baseline (F1=0.833 vs 0.721), because it could draw on background knowledge of brand product relationships that a simple rule could not access. Testing consistency across repeated runs (200 pairs, 5 runs at temperature 0.7) showed the model agreeing with itself 99.7% of the time on average, with 99% of pairs giving identical answers across all 5 runs. Using majority voting across these runs only improved F1 by 0.005, at 5 times the inference cost. These results suggest that the value of using an LLM over traditional methods depends heavily on the task. LLMs offer little advantage when strong lexical signals already exist, but a clear advantage when the task requires background knowledge, all while remaining highly consistent across repeated queries.

数据清洗大模型评估一致性品牌识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。