构建跨语言代码解析数据集，统一十种编程语言的抽象语法树结构。
MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema
- 提出通用抽象语法树架构，统一十种语言的代码结构表示。
- 包含七百万个已解析源文件，支持跨语言结构分析与学习。
- 适合研究多语言程序分析、代码表示学习的学者使用。
我们提出多语言代码解析数据集（MLCPD），一个大规模、语言无关的数据集，统一了十种主流编程语言的语法与结构表示。MLCPD 包含超过七百万个经解析的源文件，均按提出的通用抽象语法树（AST）模式标准化，支持一致的跨语言推理、结构化学习和多语言软件分析。与仅关注词法级代码或孤立解析器的现有语料库不同，MLCPD 提供层次化树结构与丰富元数据，确保无损语法覆盖与结构统一性。每条记录包含标准化模式、语言级元数据及以 Parquet 格式存储的抽象节点语义，支持高效检索。实证分析表明强跨语言结构规律——即使如 Python、Java、Go 等差异较大的语言，其语法图也可在共享模式下对齐。数据集已公开发布于 Hugging Face，配套代码库托管于 GitHub，涵盖完整数据集复现、语法编译与可视化工具，为未来跨语言表示学习与程序分析研究提供开放可复现的基础。
原文摘要 · Abstract (English)
We introduce the MultiLang Code Parser Dataset (MLCPD), a large-scale, language-agnostic dataset unifying syntactic and structural representations of code across ten major programming languages. MLCPD contains over seven million parsed source files normalized under our proposed universal Abstract Syntax Tree (AST) schema, enabling consistent cross-language reasoning, structural learning, and multilingual software analysis. Unlike existing corpora that focus purely on token-level code or isolated parsers, MLCPD provides both hierarchical tree representations and rich metadata for every file, ensuring lossless syntactic coverage and structural uniformity. Each entry includes a normalized schema, language-level metadata, and abstracted node semantics stored in Parquet format for scalable retrieval. Empirical analyses reveal strong cross-language structural regularities-demonstrating that syntactic graphs from languages as diverse as Python, Java, and Go can be aligned under a shared schema. We release the dataset publicly on Hugging Face and the accompanying codebase on GitHub, which includes complete pipelines for dataset reproduction, grammar compilation, and a visualization tool for exploring the unified AST across languages. Together, these resources establish MLCPD as an open, reproducible foundation for future research in cross-language representation learning and program analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。