研究用户如何与大模型能源管理系统协作，发现AI素养比专业知识更重要。
Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy
- 通过角色扮演实验，分析用户与GPT-4o交互的提示与决策方式。
- 95%用户使用简短提示（中位数16.2词），仅设备识别率在不同群体间有显著差异。
- 低专业用户在大模型辅助下表现接近专家，凸显大模型的平等化潜力。
本研究探讨用户建筑能源领域知识和人工智能素养如何影响人机协同建筑能源管理系统（BEMS）的有效使用。尽管已有研究关注大语言模型（LLMs）在BEMS或建筑能耗建模中的应用，但鲜有研究考察用户如何与这类系统互动。我们开展了一项系统性角色扮演实验，85名参与者与先进的生成式预训练变换器（OpenAI GPT-4o）协作，任务是识别可降低家庭能耗的前五项行为改变。收集的提示-响应数据及用户结论通过分层分析框架进行评估与打分，涵盖20个可量化指标。参与者根据自评的建筑能源知识与AI素养分为四组，采用Kruskal-Wallis H检验及事后成对比较。关键发现：多数参与者使用简洁提示（中位数16.2词），高度依赖GPT的分析能力；在20项指标中，仅有设备识别率一项存在显著组间差异（p=0.037），且由AI素养驱动，而非领域知识，表明大模型具有跨专业水平的均衡效应。本研究为大模型集成式BEMS中的人机协作机制提供了基础洞见，并推动面向人类中心的智能能源系统发展。
原文摘要 · Abstract (English)
This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。