如何让领域专家更好参与大模型设计？研究揭示关键实践与挑战。
Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study
- 通过12周实地观察，发现团队用临时方案应对数据难题。
- 专家参与评估标准制定，混合评估策略提升系统可靠性。
- 适合关注人机协作、专业领域AI落地的研究者与开发者。
大型语言模型（LLMs）正被越来越多地应用于复杂专业领域，但其实际开发与评估过程仍不清晰。本文通过为期12周的民族志研究，考察了一个教育类聊天机器人开发团队的设计与评估活动。研究人员观察了设计流程并访谈了开发者与领域专家。分析揭示了四项关键实践：为数据收集创建临时解决方案、在专家输入有限时采用增强策略、与专家共同制定评估标准、采用开发者-专家-模型混合评估方法。这些实践反映了团队在资源约束下的战略决策，凸显了领域专家在系统塑造中的核心作用。挑战包括专家参与动机与信任问题、参与式设计结构困难，以及专家知识归属与整合的疑问。论文提出未来工作流应强化人工智能素养、透明同意机制，并建立支持专家角色演化的框架。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly developed for use in complex professional domains, yet little is known about how teams design and evaluate these systems in practice. This paper examines the challenges and trade-offs in LLM development through a 12-week ethnographic study of a team building a pedagogical chatbot. The researcher observed design and evaluation activities and conducted interviews with both developers and domain experts. Analysis revealed four key practices: creating workarounds for data collection, turning to augmentation when expert input was limited, co-developing evaluation criteria with experts, and adopting hybrid expert-developer-LLM evaluation strategies. These practices show how teams made strategic decisions under constraints and demonstrate the central role of domain expertise in shaping the system. Challenges included expert motivation and trust, difficulties structuring participatory design, and questions around ownership and integration of expert knowledge. We propose design opportunities for future LLM development workflows that emphasize AI literacy, transparent consent, and frameworks recognizing evolving expert roles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。