首个基于真实故障数据的多格式文档修复基准，评估大模型修错能力。
TeXFix-Bench: An Empirically Grounded Multi-Format Benchmark for LLM-Based Document Source Repair
- 从真实崩溃日志中提炼18类故障，生成可复现的修复测试用例
- 实测大模型修复成功率仅56.7%-84.2%，且编译成功不等于内容正确
- 适合关注文档自动化修复、LLM在代码级任务表现的研究者
科学与技术写作依赖需编译的标记源（如LaTeX、Typst、Markdown），但缺失分隔符、环境不匹配、导入错误或包冲突常导致失败。现有评估使用随意编辑注入错误，缺乏实证基础。本文提出TeXFix-Bench，一个基于挖掘出的故障分类体系的多格式文档修复基准。通过对TeX Stack Exchange、GitHub提交和包文档中168个经验证的硬崩溃案例进行扎根理论研究（双人编码一致性κ=0.34），构建了18类故障的分类体系，并实现为跨三格式的48个AST感知操作符DocMut。三模型跨基准测试显示，DocMut故障比模式基变异难5.6-9.2个百分点；真实错误案例研究（88个挖掘的人工崩溃，修复成功率达67.0%）将合成数据集下限拉低。共构建10,437个实例，基于743个开源种子，评估7个大模型，采用固定零样本协议，通过提供商绑定路由，总推理成本约200美元。完整6,613实例×7模型平衡矩阵确认所有排名。绑定引擎门控带来27.5个百分点的意向治疗编译差异（56.7%-84.2%）。Typst修复难度显著高于LaTeX和Markdown。对28,129次成功编译修复的恢复原语分析表明，13.6%-18.5%的修复实质性改变了文档内容，且恢复质量排名与编译排名不一致：编译率最低的模型在成功修复中内容还原最佳。仅以编译成功衡量修复质量会严重高估效果。我们公开故障分类体系、DocMut工具及全部实验数据。
原文摘要 · Abstract (English)
Scientific and technical writing depends on markup sources that must compile: LaTeX, Typst, and Markdown pipelines fail on missing delimiters, mismatched environments, broken imports, or package conflicts. Existing document-repair evaluations inject faults with ad-hoc edits that lack an empirical fault model. We present TeXFix-Bench, a multi-format benchmark for LLM-based full-source document repair grounded in a mined fault taxonomy. A Grounded-Theory study of localized hard-crash LaTeX faults from TeX Stack Exchange, GitHub commits, and package documentation (168 verified faults, dual open coding at $κ$=0.34) yields an 18-category taxonomy instantiated as DocMut: 48 AST-aware operators across three formats. A three-model cross-benchmark shows DocMut faults are 5.6-9.2 pp harder to repair than pattern-based mutations on the same seeds, and a real-error case study (88 mined human crashes, 67.0% repair success) brackets both synthetic sets from below. We construct 10,437 instances from 743 openly licensed seeds and evaluate seven LLMs under a fixed zero-shot protocol with provider-pinned routing, collecting 48,651 attempts at about USD 200 total inference cost. A complete 6,613-instance x 7-model balanced matrix confirms all rankings. A pinned engine gate yields a 27.5-point intention-to-treat compile spread (56.7-84.2%). Typst is markedly harder than LaTeX and Markdown. A restoration oracle over 28,129 compiling repairs shows that 13.6-18.5% of compiling repairs materially alter document text, and restoration rank diverges from compile rank: the model with the lowest compile rate restores content best among its successes. Compile success alone overstates repair quality. We release the taxonomy, DocMut, and all campaign artifacts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。