用大模型解析群聊数据,自动识别集体出行决策过程。
Can large language models interpret unstructured chat data on dynamic group decision-making processes? Evidence on joint destination choice
- 设计分步提示框架,从群聊中提取决策要素
- 显性因素识别准确率高,隐性因素仍需人工判断
- 适合研究社会活动决策的学者与数据分析师
社交活动源于群体成员间的复杂联合出行决策。传统交通调查难以观测此类决策过程,而未结构化的群聊数据可提供新视角。但解读这些过程需推断显性和隐性因素,通常依赖人工标注对话以捕捉受社会文化规范影响的语境含义,耗时费力。本研究评估大语言模型(LLMs)在自动化和辅助人工标注方面的潜力,以日本集体外出就餐为例。我们设计了一种基于知识获取流程的提示框架,逐步提取关键决策因素:群体层面的餐厅选择集与最终结果、个体对各选项的偏好及其驱动属性。该结构化流程引导LLM将非结构化对话转化为描述决策因素的结构化表格数据。通过与人工标注的基准数据进行定量分析及定性错误分析,结果表明:尽管LLM能可靠识别显性决策因素,但在捕捉人类标注者轻易识别的细微隐性因素方面存在困难。研究明确了在特定情境下可信赖LLM提取结果,以及仍需人类监督的场景。这揭示了基于LLM分析非传统数据在社会活动研究中的潜力与局限。
原文摘要 · Abstract (English)
Social activities result from complex joint activity-travel decisions between group members. While observing the decision-making process of these activities is difficult via traditional travel surveys, the advent of new types of data, such as unstructured chat data, can help shed some light on these complex processes. However, interpreting these decision-making processes requires inferring both explicit and implicit factors. This typically involves the labor-intensive task of manually annotating dialogues to capture context-dependent meanings shaped by the social and cultural norms. This study evaluates the potential of Large Language Models (LLMs) to automate and complement human annotation in interpreting decision-making processes from group chats, using data on joint eating-out activities in Japan as a case study. We designed a prompting framework inspired by the knowledge acquisition process, which sequentially extracts key decision-making factors, including the group-level restaurant choice set and outcome, individual preferences of each alternative, and the specific attributes driving those preferences. This structured process guides the LLM to interpret group chat data, converting unstructured dialogues into structured tabular data describing decision-making factors. To evaluate LLM-driven outputs, we conduct a quantitative analysis using a human-annotated ground truth dataset and a qualitative error analysis to examine model limitations. Results show that while the LLM reliably captures explicit decision-making factors, it struggles to identify nuanced implicit factors that human annotators readily identified. We pinpoint specific contexts when LLM-based extraction can be trusted versus when human oversight remains essential. These findings highlight both the potential and limitations of LLM-based analysis for incorporating non-traditional data sources on social activities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。