用推理链解析人类标注差异，提升模型判断与人的一致性。
Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation
- 从推理链中提取支持/反对每种答案的语句，提升解释精度。
- 在三个数据集上优于基线方法，排序结果更贴近人类标注分布。
- 适合研究标注差异、可解释性分析的研究者使用。
近年来，经过推理优化的大语言模型（LLMs）在生成答案前会输出思维链（CoTs），这一现象为理解人类标注差异——即多位标注者对同一数据实例给出不同但合理的标签——提供了新契机。已有工作表明，基于给定答案生成解释可使模型预测更接近人类标签分布，但通常采用逆向范式。相比之下，思维链提供的是正向推理路径，可能隐含了各选项的答案理由。为此，本文提出一种新型基于LLM的流程，结合语言学驱动的论述分段器，更准确地从思维链中提取每种答案对应的支撑与反驳语句。同时，设计了一种基于排名的HLV评估框架，侧重答案排序而非精确得分，以更好比较标签分布。实验在三个数据集上验证了该方法的有效性，其性能优于直接生成法及基线模型，且排序结果与人类标注一致性更高。
原文摘要 · Abstract (English)
The recent rise of reasoning-tuned Large Language Models (LLMs)--which generate chains of thought (CoTs) before giving the final answer--has attracted significant attention and offers new opportunities for gaining insights into human label variation, which refers to plausible differences in how multiple annotators label the same data instance. Prior work has shown that LLM-generated explanations can help align model predictions with human label distributions, but typically adopt a reverse paradigm: producing explanations based on given answers. In contrast, CoTs provide a forward reasoning path that may implicitly embed rationales for each answer option, before generating the answers. We thus propose a novel LLM-based pipeline enriched with linguistically-grounded discourse segmenters to extract supporting and opposing statements for each answer option from CoTs with improved accuracy. We also propose a rank-based HLV evaluation framework that prioritizes the ranking of answers over exact scores, which instead favor direct comparison of label distributions. Our method outperforms a direct generation method as well as baselines on three datasets, and shows better alignment of ranking methods with humans, highlighting the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。