让用户用语音或文字对话方式,轻松分析表格数据并获得可视化结果。
A Multimodal Conversational Agent for Tabular Data Analysis
- 基于大模型构建多模态对话系统,支持语音输入与输出。
- 在三个数据集上完成48项任务,准确率达95.8%,生成响应低于1.7秒。
- 适合需要快速交互式数据分析的研究者与业务人员使用。
大型语言模型(LLMs)可通过与用户进行上下文感知的交互式对话,实现数据探索、可视化和解释,包括语音交互,并保持高性能。本文提出Talk2Data,一个基于多模态大模型的对话式数据探索代理。该系统允许用户通过语音或文本指令查询数据集,并以图表、表格、统计信息或语音解释的形式接收答案。系统基于大模型,整合了OpenAI Whisper自动语音识别(ASR)、Qwen-coder代码生成模型、自定义沙盒执行工具及Coqui文本转语音(TTS)库,形成智能体编排循环。相比纯文本工具,它能跨模态适应响应,支持基于数据集上下文的多轮对话。在三个数据集上对48个任务的评估显示,原型系统准确率达95.8%,模型仅生成时间低于1.7秒(不计ASR和执行时间)。五种不同规模模型(1.5B-32B)对比表明,7B模型在准确率-延迟-成本间取得最佳平衡,适合交互使用。通过在对话与代码执行间路由,受限于透明沙盒,并结合模式级上下文提示,该代理可可靠获取可操作洞察,同时确保计算过程可验证。文章还探讨了人-数据交互、大模型分析中的信任问题及未来向大规模多模态助手扩展的可能。
原文摘要 · Abstract (English)
Large language models (LLMs) can reshape information processing by handling data analysis, visualization, and interpretation in an interactive, context-aware dialogue with users, including voice interaction, while maintaining high performance. In this article, we present Talk2Data, a multimodal LLM-driven conversational agent for intuitive data exploration. The system lets users query datasets with voice or text instructions and receive answers as plots, tables, statistics, or spoken explanations. Built on LLMs, the suggested design combines OpenAI Whisper automatic speech recognition (ASR) system, Qwen-coder code generation LLM/model, custom sandboxed execution tools, and Coqui library for text-to-speech (TTS) within an agentic orchestration loop. Unlike text-only analysis tools, it adapts responses across modalities and supports multi-turn dialogues grounded in dataset context. In an evaluation of 48 tasks on three datasets, our prototype achieved 95.8% accuracy with model-only generation time under 1.7 seconds (excluding ASR and execution time). A comparison across five LLM sizes (1.5B-32B) revealed accuracy-latency-cost trade-offs, with a 7B model providing the best balance for interactive use. By routing between conversation with user and code execution, constrained to a transparent sandbox, with simultaneously grounding prompts in schema-level context, the Talk2Data agent reliably retrieves actionable insights from tables while making computations verifiable. In the article, except for the Talk2Data agent itself, we discuss implications for human-data interaction, trust in LLM-driven analytics, and future extensions toward large-scale multimodal assistants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。