用分块提示提升大模型对城市系统文本的编码准确性,媲美人类专家。
Text Chunking for Document Classification for Urban System Management using Large Language Models
- 采用分块提示法处理文本,避免大模型上下文混淆。
- GPT-4o等模型与人工编码一致性高,跨3名人类评审者显著一致。
- 适合需要高效分析城市文档的规划、管理及研究者使用。
城市系统管理依赖复杂文本资料,需进行编码与分析以制定需求并评估建成环境绩效。本文研究大语言模型(LLM)在定性编码中的应用,旨在降低资源消耗的同时保持与人类相当的可靠性。面对人力成本高、评估偏差、准确性和一致性差等问题,本研究利用GPT-4o、GPT-4o-mini和o1-mini模型,通过全篇分析与分块分析两种提示方法,对10份案例文档中17项数字孪生特征的存在情况进行演绎式编码。结果表明,分块方法显著提升了模型与人工编码的一致性;在加入两名大模型作为额外评判者后,整体评估者间达成统计学上显著一致,证明文本分析可受益于大模型辅助。研究揭示了大模型可能遵循人类记忆编码机制,全篇分析易引入多重含义。新贡献在于评估OpenAI GPT系列模型表现,并提出分块提示策略,有效缓解上下文聚合偏差。
原文摘要 · Abstract (English)
Urban systems are managed using complex textual documentation that need coding and analysis to set requirements and evaluate built environment performance. This paper contributes to the study of applying large-language models (LLM) to qualitative coding activities to reduce resource requirements while maintaining comparable reliability to humans. Qualitative coding and assessment face challenges like resource limitations and bias, accuracy, and consistency between human evaluators. Here we report the application of LLMs to deductively code 10 case documents on the presence of 17 digital twin characteristics for the management of urban systems. We utilize two prompting methods to compare the semantic processing of LLMs with human coding efforts: whole text analysis and text chunk analysis using OpenAI's GPT-4o, GPT-4o-mini, and o1-mini models. We found similar trends of internal variability between methods and results indicate that LLMs may perform on par with human coders when initialized with specific deductive coding contexts. GPT-4o, o1-mini and GPT-4o-mini showed significant agreement with human raters when employed using a chunking method. The application of both GPT-4o and GPT-4o-mini as an additional rater with three manual raters showed statistically significant agreement across all raters, indicating that the analysis of textual documents is benefited by LLMs. Our findings reveal nuanced sub-themes of LLM application suggesting LLMs follow human memory coding processes where whole-text analysis may introduce multiple meanings. The novel contributions of this paper lie in assessing the performance of OpenAI GPT models and introduces the chunk-based prompting approach, which addresses context aggregation biases by preserving localized context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。