评测大模型生成LaTeX代码的能力，发现越复杂越容易出错。
TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
- 设计多层级任务，用自然语言指令生成科学文档LaTeX代码
- 开放模型如DeepSeek在复杂任务中表现接近闭源模型
- 格式和包引用错误普遍，暴露训练数据多样性不足
LaTeX因其精准排版能力成为科学文献撰写的标准工具。大语言模型（LLMs）为研究者通过自然语言生成可发表文档提供了可能，但现有评测体系完全缺失对这一能力的评估。为此，我们提出TeXpert基准数据集，包含针对科学文档不同组件、分多难度层级的自然语言提示，用于生成LaTeX代码。我们对开源与闭源模型进行了深度评估，发现：在标准基准上表现优异的LLMs在LaTeX生成任务中准确率显著下降，且随任务复杂度增加性能滑坡明显；开源模型如DeepSeek v3和DeepSeek Coder在该任务中表现强劲，媲美闭源模型；格式错误和包引用错误异常普遍，表明多数LLM训练数据缺乏多样化的LaTeX实例。相关数据集、代码与评估结果已公开于https://github.com/knowledge-verse-ai/TeXpert。
原文摘要 · Abstract (English)
LaTeX's precision and flexibility in typesetting have made it the gold standard for the preparation of scientific documentation. Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material using LaTeX with natural language instructions, yet current benchmarks completely lack evaluation of this ability. By introducing TeXpert, our benchmark dataset with natural language prompts for generating LaTeX code focused on components of scientific documents across multiple difficulty levels, we conduct an in-depth analysis of LLM performance in this regard and identify frequent error types. Our evaluation across open and closed-source LLMs highlights multiple key findings: LLMs excelling on standard benchmarks perform poorly in LaTeX generation with a significant accuracy drop-off as the complexity of tasks increases; open-source models like DeepSeek v3 and DeepSeek Coder strongly rival closed-source counterparts in LaTeX tasks; and formatting and package errors are unexpectedly prevalent, suggesting a lack of diverse LaTeX examples in the training datasets of most LLMs. Our dataset, code, and model evaluations are available at https://github.com/knowledge-verse-ai/TeXpert.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。