用标准化提示框架提升大模型在语料分析中的可复现性
TACOMORE: Exploring a replicable prompting protocol for LLM-assisted corpus analysis
- 设计TACOMORE框架，基于任务、上下文、模型和可复现性四原则
- 在新冠研究摘要语料上验证，结构化提示提升分析准确性和重复性
- 适合需要可复现语料分析结果的研究者，尤其关注人类校验必要性
随着语料规模扩大，研究人员面临方法瓶颈：计算工具虽能快速统计数十亿词，但定性解读仍依赖耗时的人工。大语言模型（LLMs）有望自动化该过程，但其黑箱特性与缺乏可复现性常阻碍应用。本文提出TACOMORE，一个结构化提示框架，将随意的AI交互转化为标准化语言学协议。该框架基于任务、上下文、模型和可复现性四大原则，引导LLM从泛化概率预测转向基于目标语料共现模式的推理。我们在新冠研究摘要开放语料上，对关键词、搭配词和语境实例三类核心任务进行了测试。评估了三种LLM，结果显示结构化提示可提升准确率与可复现性，但幻觉问题仍存在。本研究为LLM在语料学中的角色提供批判性视角，强调其作为辅助工具的潜力，同时突出人类验证的不可替代性。
原文摘要 · Abstract (English)
As corpus linguistics continues to scale, researchers are facing a growing methodological bottleneck: while computational tools can easily count billions of words, the qualitative interpretation of these data remains a slow and labor-intensive human task. Large Language Models (LLMs) offer a promising way to automate this process, yet their integration into the field is often hindered by concerns over black-box unpredictability and a lack of replicability. This study introduces TACOMORE, a structured prompting framework designed to transform ad-hoc AI interactions into a standardized linguistic protocol. Built upon four foundational principles (Task, Context, Model, and Replicability), the framework guides LLMs to move beyond generic probability prediction to anchoring their reasoning in the specific co-occurrence patterns of a target corpus. We applied this framework to three core corpus tasks, i.e., the analysis of keywords, collocates, and concordances, using an open corpus of COVID-19 research abstracts. After testing three LLMs, we found that while structured prompting improves accuracy and replicability, inherent limitations regarding hallucination persist. This research offers a critical lens into the role of LLMs in corpus linguistics, highlighting their potential as complementary tools while emphasizing the irreplaceable role of human validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。