arXiv:2502.00682cs.HCcs.AI2025-02被引 2

不同来源的AI指导影响用户数据分析体验,效果因人而异。

Guidance Source Matters: How Guidance from AI, Expert, or a Group of Analysts Impacts Visual Data Preparation and Analysis

  • 对比AI、专家、群体分析师和无来源指导的效果
  • 相同质量下,用户对AI指导使用更多且后悔感更强
  • 适合研究人机协作中信任与反馈机制的设计

生成式AI的进步推动了以协作者和助手为代表的AI工具提供更优指导,尤其在数据分析阶段。然而,现有研究尚未考察指导来源的感知有效性及其对用户认知与使用行为的影响。本文探究用户是否认为所有指导来源等效,重点关注四种来源:(i) AI、(ii) 人类专家、(iii) 一组分析师、(iv) 无来源指导(即不标明出处),后者用于隔离并比较特定来源的影响。设计五组被试间实验,每组对应一种指导来源,另设无指导组作为基线。研究在自研数据准备与分析工具中展开,要求用户从陌生数据集中选择相关属性以撰写业务报告。根据分组,用户可请求指导,系统据此提供属性建议。为确保内部效度,各来源指导质量保持一致。通过多维度使用与感知指标,统计检验五个预注册假设,并报告额外分析。结果发现,指导来源显著影响用户行为,但并不符合常识预期:用户在分析不同阶段对指导的使用方式不同,即便指导质量相同,仍表现出不同程度的后悔感。特别地,使用AI指导的用户报告了更高的任务后收益感与后悔感。

原文摘要 · Abstract (English)

The progress in generative AI has fueled AI-powered tools like co-pilots and assistants to provision better guidance, particularly during data analysis. However, research on guidance has not yet examined the perceived efficacy of the source from which guidance is offered and the impact of this source on the user's perception and usage of guidance. We ask whether users perceive all guidance sources as equal, with particular interest in three sources: (i) AI, (ii) human expert, and (iii) a group of human analysts. As a benchmark, we consider a fourth source, (iv) unattributed guidance, where guidance is provided without attribution to any source, enabling isolation of and comparison with the effects of source-specific guidance. We design a five-condition between-subjects study, with one condition for each of the four guidance sources and an additional (v) no-guidance condition, which serves as a baseline to evaluate the influence of any kind of guidance. We situate our study in a custom data preparation and analysis tool wherein we task users to select relevant attributes from an unfamiliar dataset to inform a business report. Depending on the assigned condition, users can request guidance, which the system then provides in the form of attribute suggestions. To ensure internal validity, we control for the quality of guidance across source-conditions. Through several metrics of usage and perception, we statistically test five preregistered hypotheses and report on additional analysis. We find that the source of guidance matters to users, but not in a manner that matches received wisdom. For instance, users utilize guidance differently at various stages of analysis, including expressing varying levels of regret, despite receiving guidance of similar quality. Notably, users in the AI condition reported both higher post-task benefit and regret.

人机协作数据分析指导源用户体验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。