让不懂统计的用户也能问出无偏见的数据分析问题。
VeriMinder: Mitigating Analytical Vulnerabilities in NL2SQL
- 用上下文感知框架识别分析中的认知偏差
- 82.5%用户认为系统显著提升分析质量
- 适合需要避免错误提问的研究者和业务人员
使用自然语言数据库接口（NLIDB）的应用系统已使数据分析民主化，但同时也带来挑战：缺乏统计背景的用户可能提出带有偏见的分析问题。尽管文本转SQL的准确性研究较多，但对分析性问题中认知偏差的治理仍不充分。本文提出VeriMinder，一个交互式系统，用于检测并缓解此类分析漏洞。其创新包括：(1) 针对特定分析场景的上下文语义偏差映射框架；(2) 基于‘难变性原则’的可操作分析框架，引导用户系统化分析；(3) 优化的基于大模型的提示生成系统，通过多候选、批评反馈与自我反思实现高质量任务定制提示。用户测试表明，82.5%参与者认为系统显著提升了分析质量。在对比评估中，VeriMinder在分析具体性、全面性和准确性上均至少优于其他方法20%。系统已作为Web应用上线，代码与提示集以MIT开源，支持社区进一步研究与应用。
原文摘要 · Abstract (English)
Application systems using natural language interfaces to databases (NLIDBs) have democratized data analysis. This positive development has also brought forth an urgent challenge to help users who might use these systems without a background in statistical analysis to formulate bias-free analytical questions. Although significant research has focused on text-to-SQL generation accuracy, addressing cognitive biases in analytical questions remains underexplored. We present VeriMinder, https://veriminder.ai, an interactive system for detecting and mitigating such analytical vulnerabilities. Our approach introduces three key innovations: (1) a contextual semantic mapping framework for biases relevant to specific analysis contexts (2) an analytical framework that operationalizes the Hard-to-Vary principle and guides users in systematic data analysis (3) an optimized LLM-powered system that generates high-quality, task-specific prompts using a structured process involving multiple candidates, critic feedback, and self-reflection. User testing confirms the merits of our approach. In direct user experience evaluation, 82.5% participants reported positively impacting the quality of the analysis. In comparative evaluation, VeriMinder scored significantly higher than alternative approaches, at least 20% better when considered for metrics of the analysis's concreteness, comprehensiveness, and accuracy. Our system, implemented as a web application, is set to help users avoid "wrong question" vulnerability during data analysis. VeriMinder code base with prompts, https://reproducibility.link/veriminder, is available as an MIT-licensed open-source software to facilitate further research and adoption within the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。