用AI辅助可视化,帮专家从多模态追踪数据中提炼有意义的上下文洞察。
Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM
- 结合LLM与可视化,支持专家在循环中验证和探索行为洞察。
- 通过三轮用户研究,发现专家需在原始数据与AI推断间反复比对。
- 为未来健康传感数据系统设计提供可复用的交互范式参考。
被动式追踪方法(如手机与可穿戴设备传感)已成为现代普适计算研究中监测人类行为的主要手段。尽管机器学习在将原始传感器数据转化为瞬时行为识别(如体力活动检测)方面取得显著进展,但在将这些传感流转化为有意义、高层级且具备上下文意识的洞察方面仍存在明显差距,而这正是许多应用(如总结个人日常规律)所必需的。为弥合这一鸿沟,专家在真实世界研究中常需进行上下文驱动的情境理解过程,但该过程往往依赖手动操作,即使对经验丰富的研究者也颇具挑战性。我们通过对21位专家开展三轮用户研究,探索情境理解中的挑战并提出解决方案。基于以人为本的设计流程,我们开发了名为Vital Insight(VI)的原型系统,该系统利用大语言模型实现人机协同推理与可视化,以支持多模态被动传感数据(来自智能手机与可穿戴设备)的情境理解。通过将原型作为技术探针,观察专家与系统的互动,我们提炼出一个专家情境理解模型,揭示专家如何在直接数据表示与AI支持的推断之间来回切换,以探索、质疑并验证洞察。在此迭代过程中,我们进一步归纳并讨论了一系列面向未来人工智能增强型可视化系统的架构设计启示。
原文摘要 · Abstract (English)
Passive tracking methods, such as phone and wearable sensing, have become dominant in monitoring human behaviors in modern ubiquitous computing studies. While there have been significant advances in machine-learning approaches to translate periods of raw sensor data to model momentary behaviors, (e.g., physical activity recognition), there still remains a significant gap in the translation of these sensing streams into meaningful, high-level, context-aware insights that are required for various applications (e.g., summarizing an individual's daily routine). To bridge this gap, experts often need to employ a context-driven sensemaking process in real-world studies to derive insights. This process often requires manual effort and can be challenging even for experienced researchers due to the complexity of human behaviors. We conducted three rounds of user studies with 21 experts to explore solutions to address challenges with sensemaking. We follow a human-centered design process to identify needs and design, iterate, build, and evaluate Vital Insight (VI), a novel, LLM-assisted, prototype system to enable human-in-the-loop inference (sensemaking) and visualizations of multi-modal passive sensing data from smartphones and wearables. Using the prototype as a technology probe, we observe experts' interactions with it and develop an expert sensemaking model that explains how experts move between direct data representations and AI-supported inferences to explore, question, and validate insights. Through this iterative process, we also synthesize and discuss a list of design implications for the design of future AI-augmented visualization systems to better assist experts' sensemaking processes in multi-modal health sensing data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。