arXiv:2601.09905cs.SEcs.CL2026-01

用大模型自我反思提升文本标注准确率,降低错误率。

Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique

  • 第一阶段用大模型按设计好的编码本标注,第二阶段由另一模型重审并修正错误。
  • 经二次审核后F1得分提升0.04至0.25,部分差的编码从0.52提至0.69。
  • 适合需要高精度标注但原始模型误差高的研究者,尤其适用于有明确错误模式的场景。

大语言模型可高效处理大规模文本定性编码,但零样本和少样本分类器仍会产生大量错误,即使使用精心设计的提示。本文提出一种简单通用的两阶段流程:第一阶段由大模型依据人工设计、适配大模型的编码本进行标注;第二阶段由另一个大模型对每个阳性标签进行自我反思,重新阅读原文与首阶段理由,作出最终判断。在来自Apache软件基金会项目评估讨论的3000封高信息量邮件上测试六种定性编码。人工审计360个阳性标注(每种编码60段)发现,首阶段模型误报率高达8%至54%,尽管测试F1为0.74至1.00。经第二阶段自我反思重标后,F1提升0.04至0.25,两个表现较差的编码分别从0.52和0.55提升至0.69和0.79。人工分析识别出两类常见错误:违背编码定义的误解,以及将项目评估标准的讨论误认为决策依据。针对特定错误模式设计的批评条款效果显著,可在测试中持续优化,实现对首阶段模型理解的编码本适配。通过优先召回首阶段标注,再以轻量计算实现精确控制,该方法可在人类引导下嵌入现有大模型辅助标注流程,有效降噪并挽救原本不可用的分类器。

原文摘要 · Abstract (English)

Large language models (LLMs) allow for sophisticated qualitative coding of large datasets, but zero- and few-shot classifiers can produce an intolerable number of errors, even with careful, validated prompting. We present a simple, generalizable two-stage workflow: an LLM applies a human-designed, LLM-adapted codebook; a secondary LLM critic performs self-reflection on each positive label by re-reading the source text alongside the first model's rationale and issuing a final decision. We evaluate this approach on six qualitative codes over 3,000 high-content emails from Apache Software Foundation project evaluation discussions. Our human-derived audit of 360 positive annotations (60 passages by six codes) found that the first-line LLM had a false-positive rate of 8% to 54%, despite F1 scores of 0.74 and 1.00 in testing. Subsequent recoding of all stage-one annotations via a second self-reflection stage improved F1 by 0.04 to 0.25, bringing two especially poor performing codes up to 0.69 and 0.79 from 0.52 and 0.55 respectively. Our manual evaluation identified two recurrent error classes: misinterpretation (violations of code definitions) and meta-discussion (debate about a project evaluation criterion mistaken for its use as a decision justification). Code-specific critic clauses addressing observed failure modes were especially effective with testing and refinement, replicating the codebook-adaption process for LLM interpretation in stage-one. We explain how favoring recall in first-line LLM annotation combined with secondary critique delivers precision-first, compute-light control. With human guidance and validation, self-reflection slots into existing LLM-assisted annotation pipelines to reduce noise and potentially salvage unusable classifiers.

定性编码大模型自我反思标注优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。