用大模型自动生成科学数据分析代码，但需改进才能可靠使用。
Toward Automated and Trustworthy Scientific Analysis and Visualization with LLM-Generated Code
- 用自然语言生成代码，结合真实科研任务设计评测基准。
- 无干预下代码执行成功率低，主要因提示模糊和领域理解不足。
- 提出三种增强方法，提升代码可执行性，适合科研自动化工具开发者。
随着现代科学日益数据密集，高效分析与可视化大规模复杂数据集对加速发现至关重要。然而，许多领域科学家缺乏编程能力，难以构建定制化分析流程，阻碍了及时洞察。大语言模型（LLMs）通过从自然语言描述生成可执行代码，提供了一种潜在解决方案。本文研究开源LLMs在自主生成科学数据分析与可视化Python脚本时的可信度。我们构建了一个反映真实科研任务的领域导向提示基准套件，并系统评估生成代码的可执行性与正确性。结果表明，在无人干预下，代码可靠性有限，频繁失败源于提示模糊及模型对领域上下文理解不足。为此，我们设计并评估了三种互补策略：数据感知提示消歧、检索增强提示优化与迭代错误修复。这些方法显著提升了执行成功率与输出质量，但仍需进一步优化。本工作揭示了LLM驱动科学工作流自动化的潜力与当前局限，并提出了可复用的技术方案与基准，助力构建更包容、易用且可信的AI辅助科研工具。
原文摘要 · Abstract (English)
As modern science becomes increasingly data-intensive, the ability to analyze and visualize large-scale, complex datasets is critical to accelerating discovery. However, many domain scientists lack the programming expertise required to develop custom data analysis workflows, creating barriers to timely and effective insight. Large language models (LLMs) offer a promising solution by generating executable code from natural language descriptions. In this paper, we investigate the trustworthiness of open-source LLMs in autonomously producing Python scripts for scientific data analysis and visualization. We construct a benchmark suite of domain-inspired prompts that reflect real-world research tasks and systematically evaluate the executability and correctness of the generated code. Our findings show that, without human intervention, the reliability of LLM-generated code is limited, with frequent failures caused by ambiguous prompts and the models' insufficient understanding of domain-specific contexts. To address these challenges, we design and assess three complementary strategies: data-aware prompt disambiguation, retrieval-augmented prompt enhancement, and iterative error repair. While these methods significantly improve execution success rates and output quality, further refinement is needed. This work highlights both the promise and current limitations of LLM-driven automation in scientific workflows and introduces actionable techniques and a reusable benchmark for building more inclusive, accessible, and trustworthy AI-assisted research tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。