首个评估科学演示代码交互生成能力的基准,融合程序测试与视觉比对。
InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generation
- 结合程序测试与视觉比对,验证代码交互逻辑与渲染效果。
- 覆盖5个科学领域,含单元测试、参考快照和检查清单。
- 评测30个主流模型,揭示知识与交互编码融合的短板。
大型语言模型(LLMs)正日益能够从自然语言指令生成完整应用,为科学与教育带来新机遇。在这些领域,交互式科学演示对于解释概念、支持新型教学法和展示研究成果尤为关键。生成此类演示需模型兼具准确的科学知识与实现响应用户操作的交互前端代码的能力。现有基准多仅评估知识问答或静态网页代码生成,无法涵盖这一综合能力。为此,我们设计了一种混合框架,结合程序化功能测试以严格验证交互逻辑,以及基于视觉的定性测试以评估输出渲染结果与参考快照的一致性。在此基础上,我们提出InteractScience基准,包含跨五个科学领域的大量精心设计问题,每个问题配有序列化测试用例、参考快照和检查清单。我们评估了30个主流开源与闭源LLM,结果凸显其在整合领域知识与交互前端编码方面的持续不足。本工作将InteractScience定位为首个可自动衡量真实交互操作下综合能力的基准,为提升可靠且具教育价值的科学演示代码生成奠定基础。所有代码与数据均公开于https://github.com/open-compass/InteractScience。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly capable of generating complete applications from natural language instructions, creating new opportunities in science and education. In these domains, interactive scientific demonstrations are particularly valuable for explaining concepts, supporting new teaching methods, and presenting research findings. Generating such demonstrations requires models to combine accurate scientific knowledge with the ability to implement interactive front-end code that behaves correctly and responds to user actions. This capability goes beyond the scope of existing benchmarks, which typically evaluate either knowledge question answering without grounding in code or static web code generation without scientific interactivity. To evaluate this integrated ability, we design a hybrid framework that combines programmatic functional testing to rigorously verify interaction logic with visually-grounded qualitative testing to assess rendered outputs against reference snapshots. Building on this framework, we present InteractScience, a benchmark consisting of a substantial set of carefully designed questions across five scientific domains, each paired with unit tests, reference snapshots, and checklists. We evaluate 30 leading open- and closed-source LLMs and report results that highlight ongoing weaknesses in integrating domain knowledge with interactive front-end coding. Our work positions InteractScience as the first benchmark to automatically measure this combined capability with realistic interactive operations, providing a foundation for advancing reliable and educationally useful scientific demonstration code generation. All code and data are publicly available at https://github.com/open-compass/InteractScience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。