用AI自动重用神经科学数据,解决格式混乱难题。
Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

- 让AI读代码和论文,自动加载并转换不同格式的神经数据。
- 多数AI能完成单步任务,但难以端到端无错处理全流程。
- 强调人工介入必要性,提出适配AI时代的数据共享规范。
神经科学数据分散在不同实验室、格式和实验范式中,重用常需大量手动工作。核心障碍在于需理解各具特色的数据格式。虽有通用格式提议,但灵活性与自解释性难兼顾:灵活的格式描述不足,描述性强的格式又需持续维护,多数实验室难以承担。本研究考察了代理型AI在此任务中的表现,选取8篇包含数据与代码的大规模小鼠神经群体记录论文,涵盖多种记录模态、行为范式及格式(如NWB、专用API、通用Python/MATLAB文件)。我们向代理提供数据、代码和论文,要求其完成从神经活动解码任务变量或行为变量的下游任务。通用编程代理在各子任务中表现良好,但极少能实现全链路无错误解决方案。我们分析了代理出错类型及引发错误的数据特征,并提出面向代理时代的数据共享最佳实践。此外发现,代理作为判别者不可靠,尤其缺乏真实参照时,因此人机协同编程仍不可或缺。
原文摘要 · Abstract (English)
Neuroscience data are highly fragmented across labs, formats, and experimental paradigms, and reuse often requires substantial manual effort. A persistent roadblock to data reuse and integration is the need to decipher bespoke and diverse data formatting choices. Common data formats have been proposed in response, but the field continues to struggle with a fundamental tension: formats flexible enough to accommodate diverse experiments are rarely descriptive enough to be self-explanatory, and sufficiently descriptive formats demand detailed documentation and curation effort that few labs can sustain. Agentic AI is a natural candidate to solve this problem: LLMs read code and text faster and with sustained attention to the low-level details humans tend to skim over. To measure how well agentic AI performs on this task, we selected eight recent papers studying large-scale mouse neural population recordings that shared both data and code, spanning diverse recording modalities, behavioral paradigms, and dataset formats (e.g., NWB, specialized APIs, and general-purpose Python or MATLAB files). We provided agents with the data, code, and paper, and prompted them to load, understand, and reformat the data for a common downstream task: training a decoder from neural activity to task or behavioral variables. General-purpose coding agents commonly used by scientists performed well on each sub-task, but rarely strung together a fully error-free end-to-end solution. We characterize the types of mistakes agents made and the dataset properties that elicited them, and propose data-sharing best practices for the agentic-AI era. We further find that agents-as-judges are unreliable at catching errors, especially without ground-truth references, so interactive, human-in-the-loop coding remains necessary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。