arXiv:2409.13879cs.CL2024-09EMNLP被引 25

构建政治访谈问答清晰度数据集与分类体系,助力大模型理解模糊回应。

"I Never Said That": A dataset, taxonomy and baselines on response clarity classification

  • 提出两级清晰度分类框架,涵盖信息量与回避技巧
  • 基于政治访谈构建标注数据集,含多轮问答对
  • 结合大模型与人工标注,建立新基准测试

公共言论中的模棱两可现象在政治学和政治访谈分析中已有深入研究。受相关理论启发,本文聚焦于从政治访谈中提取的问题回答的清晰度问题,利用大语言模型(LLMs)与人类专家能力,提出一种新的清晰度分类体系,并构建相应的清晰度分类数据集。该数据集包含从政治访谈中抽取的问答对,经人工标注并按新提出的两级分类体系进行标注:高层级关注回答对问题的信息提供程度,低层级则细分为多种回避技巧类别。通过结合ChatGPT与人工标注者,收集、验证并标注了多个问答对,用于支持新的回答清晰度任务。我们进行了详尽的分析,并在不同模型架构、规模及微调方法下开展实验,以获得深入洞察并建立该任务的新基线。

原文摘要 · Abstract (English)

Equivocation and ambiguity in public speech are well-studied discourse phenomena, especially in political science and analysis of political interviews. Inspired by the well-grounded theory on equivocation, we aim to resolve the closely related problem of response clarity in questions extracted from political interviews, leveraging the capabilities of Large Language Models (LLMs) and human expertise. To this end, we introduce a novel taxonomy that frames the task of detecting and classifying response clarity and a corresponding clarity classification dataset which consists of question-answer (QA) pairs drawn from political interviews and annotated accordingly. Our proposed two-level taxonomy addresses the clarity of a response in terms of the information provided for a given question (high-level) and also provides a fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous responses (lower-level). We combine ChatGPT and human annotators to collect, validate and annotate discrete QA pairs from political interviews, to be used for our newly introduced response clarity task. We provide a detailed analysis and conduct several experiments with different model architectures, sizes and adaptation methods to gain insights and establish new baselines over the proposed dataset and task.

自然语言理解政治对话清晰度评估大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。