arXiv:2508.13124cs.CLcs.AI2025-08被引 1

发现并量化大模型在客服摘要中的细微偏见,揭示其普遍存在且不因模型大小而改变。

Spot the BlindSpots: Systematic Identification and Quantification of Fine-Grained LLM Biases in Contact Center Summaries

  • 构建15类运营偏见分类体系,用大模型做零样本分类识别偏见
  • 发现20个模型在2500条真实通话中均存在系统性偏见
  • 适合关注生成质量与公平性的企业级AI应用开发者

摘要:摘要生成是客服中心的核心应用,大型语言模型(LLMs)每天生成数百万条通话记录的摘要。尽管表面质量良好,但尚不清楚这些模型是否系统性地忽略或过度关注某些文本特征,从而引入偏见。现有研究多关注社会性与位置性偏见,而对客服场景特有的“运营偏见”尚未探索。为此,我们提出盲点检测框架BlindSpot,基于15种运营偏见维度(如口吃、发言者、话题等),识别并量化此类偏见。BlindSpot利用大模型作为零样本分类器,为每一对通话原文及其摘要生成各偏见维度的类别分布,并通过两个指标衡量偏见:保真度差距(源与目标分布间的JS散度)和覆盖率(源标签遗漏比例)。我们在2500条真实通话及20个不同规模与家族(如GPT、Llama、Claude)的模型生成的摘要上进行了实证研究。结果表明,偏见具有系统性,所有被测模型均存在,无论规模或类型。

原文摘要 · Abstract (English)

Abstractive summarization is a core application in contact centers, where Large Language Models (LLMs) generate millions of summaries of call transcripts daily. Despite their apparent quality, it remains unclear whether LLMs systematically under- or over-attend to specific aspects of the transcript, potentially introducing biases in the generated summary. While prior work has examined social and positional biases, the specific forms of bias pertinent to contact center operations - which we term Operational Bias - have remained unexplored. To address this gap, we introduce BlindSpot, a framework built upon a taxonomy of 15 operational bias dimensions (e.g., disfluency, speaker, topic) for the identification and quantification of these biases. BlindSpot leverages an LLM as a zero-shot classifier to derive categorical distributions for each bias dimension in a pair of transcript and its summary. The bias is then quantified using two metrics: Fidelity Gap (the JS Divergence between distributions) and Coverage (the percentage of source labels omitted). Using BlindSpot, we conducted an empirical study with 2500 real call transcripts and their summaries generated by 20 LLMs of varying scales and families (e.g., GPT, Llama, Claude). Our analysis reveals that biases are systemic and present across all evaluated models, regardless of size or family.

大模型偏见客服摘要偏见检测生成质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。