让大模型做定性分析可审计,确保每一步都有据可查。
Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis

- 用可追溯的流程结合大模型与确定性控制生成编码和主题。
- 生成的编码覆盖接近人工水平,分析理由评分高,主题更精炼。
- 适合需要透明、可复现分析结果的研究者,尤其关注方法可信度。
大型语言模型(LLMs)为扩展定性分析提供了新可能,但现有应用在如何将定性方法转化为计算流程方面常缺乏方法透明性。本文提出一种可审计且隐私保护的归纳式与潜在主题分析(TA)的计算实现方案。首先,基于TA的方法论要求和基于LLM推理的条件,提炼出五项设计原则:保持解释语境、维护经验材料与分析输出间的可追溯关系、显式表示分析构念与推理过程、限制LLM推理仅用于解释任务、支持隐私保护的本地部署。其次,提出一个两阶段工作流原型,通过结合解释性LLM推理与确定性程序控制,生成代码、分析理由、主题及主题描述,并保留与原始材料的明确关联。第三,提出一个评估框架,结合结构比较与人工主导的TA、独立专家对分析质量的评估。在半结构化丹麦访谈转录本上进行评估,结果显示该工作流生成的代码覆盖率与人工标注基本相当,分析理由评分高,同时生成更紧凑的主题结构——主题数量更少、范围更广。研究证明了通过模块化工作流实现可审计的LLM支持主题分析的可行性,该流程可扩展至更大数据集,兼容不同LLM,并支持跨研究领域迁移,领域适配主要依赖提示策略调整。
原文摘要 · Abstract (English)
Large Language Models (LLMs) offer new possibilities for scaling qualitative analysis, but existing applications often provide limited methodological transparency regarding how qualitative methods are translated into computational procedures. This paper presents an auditable and privacy-preserving computational operationalization of inductive and latent Thematic Analysis (TA). This paper first derives five design principles from the methodological requirements of TA and the conditions introduced by LLM-based inference: preserving interpretative context, maintaining traceable relationships between empirical material and analytical outputs, representing analytical constructs and reasoning explicitly, constraining LLM inference to interpretative tasks, and enabling privacy-preserving local deployment. Second, it presents a proof-of-concept for a two-phase workflow that operationalizes these principles by combining interpretative LLM inference with deterministic procedural control to generate codes, analytical justifications, themes, and theme descriptions while preserving explicit links to the source material. Third, it proposes an evaluation framework combining structural comparison with human-led TA and independent expert assessment of analytical quality. The evaluation is conducted on semi-structured Danish interview transcripts. and the results shows that the workflow produces code-level outputs with coverage broadly comparable to human annotations and highly rated analytical justifications, while generating a more compressed thematic structure characterized by fewer and broader themes. The findings demonstrate the feasibility of auditable LLM-supported TA through a modular workflow designed to scale to larger datasets, accommodate different LLMs, and support transfer across research domains, with domain adaptation primarily requiring adjustments to the prompting strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。