用AI聊天机器人提升网络问卷深度，兼顾数据质量与可扩展性
AI-Assisted Conversational Interviewing: Effects on Data Quality and Respondent Experience
- 用大语言模型动态追问并实时编码开放题回答
- 开放题回答更详细，但用户体验略有下降
- 无需定制微调即可实现中等水平的实时编码
标准化问卷虽易扩展但缺乏深度，而对话式访谈虽能提升回答质量却难以规模化。本研究提出一种AI辅助对话式访谈框架，通过网页实验验证：1800名参与者被随机分配至使用大语言模型（LLMs）的AI聊天机器人，该机器人可动态追问以获取补充信息，并对开放题进行交互式编码。评估显示，尽管因应答者顺从偏差导致误报率略高，但未经过特定调查微调的AI聊天机器人仍能在实时编码中表现中等水平。开放题回答内容更丰富、信息量更大，但受访者体验略有下降。结果表明，基于大语言模型的聊天机器人在增强网络调查中的开放式数据收集方面具有可行性。
原文摘要 · Abstract (English)
Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for AI-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to AI 'chatbots' which use large language models (LLMs) to dynamically probe respondents for elaboration and interactively code open-ended responses to fixed questions developed by human researchers. We assessed the AI chatbot's performance in terms of coding accuracy, response quality, and respondent experience. Our findings reveal that AI chatbots perform moderately well in live coding even without survey-specific fine-tuning, despite slightly inflated false positive errors due to respondent acquiescence bias. Open-ended responses were more detailed and informative, but this came at a slight cost to respondent experience. Our findings highlight the feasibility of using AI methods such as chatbots enhanced by LLMs to enhance open-ended data collection in web surveys.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。