评测大模型在晶体材料结构修改中的空间推理能力，发现其仍依赖人类协作。
AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
- 构建10种基础操作的基准测试，涵盖四大建模类别。
- 复杂操作如旋转成功率低于12%，显示空间推理短板。
- 适合开发结构感知模型或智能科研助手的研究者使用。
大型语言模型（LLMs）在科学研究中展现出巨大潜力，可支持从知识检索到性质预测的任务。现有科学基准多聚焦于感知或知识类任务，忽视了建模这一科研核心环节。在材料科学中，原子结构的构建与操作是最具创造性且自动化程度最低的步骤之一。本文提出AtomWorld，一个用于评估LLMs在结构修改能力上的基准，包含十种基础操作和四大常见建模类别，支持可验证的评估指标。实验表明，Claude Opus 4.6整体表现最佳，但随着建模复杂度提升，成功率显著下降，尤其在涉及复杂空间关系的操作（如旋转）中，成功率低于12%。结果表明，当前大模型更适合作为材料结构建模的协作者，而非完全自主的科研代理。此外，AtomWorld还可作为未来结构感知模型（如强化学习、智能体方法）的研发试验场。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown promising potential in scientific research, enabling tasks ranging from knowledge retrieval to property prediction. Existing science benchmarks mainly focus on perceptual or knowledge-based tasks, largely ignoring the modelling tasks, a fundamental starting point for any real scientific research. For materials science, constructing and manipulating atomic structures is one of the most creative and least automated steps. In this work, we introduce AtomWorld, a benchmark designed to evaluate the abilities of LLMs on structure modifications. The benchmark includes ten fundamental actions under four widely used modelling categories, enabling verifiable evaluation metrics. We find that Claude Opus 4.6 generally performs the best. While the success rate decreases markedly with increasing modelling complexity, with particularly low success rates (below 12\% for rotation) for operations involving complex spatial relations. Our results suggest that contemporary LLMs are better suited as copilots for materials structure modelling rather than fully unsupervised autonomous scientific agents. Beyond evaluation, AtomWorld also serves as a testbed and playground for developing future structure-aware models, including reinforcement learning and agentic approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。