让数学形式化翻译错误可被定位和修复,提升自动推理系统可靠性。
FormalRx: Rectify and eXamine Semantic Failures in Autoformalization

- 构建28类错误分类体系,按优先级分解翻译失败原因。
- 诊断模型在测试集上错误定位准确率达75%,修正成功率73%。
- 适合关注数学推理系统可解释性与性能优化的研究者。
自动形式化中的语义对齐对形式化数学推理至关重要。然而现有评估仅提供不透明的二元判断或标量分数,无法揭示翻译失败的具体位置与原因,严重制约人类理解与系统改进。为此,我们提出 FormalRx,一个全面的诊断评估框架,将自动形式化评估从黑箱判断转变为可操作反馈。核心是 SCI 错误分类体系,将错误细分为28个类别并设定严格优先级。基于此,FormalRx 提供四类诊断能力:对齐判断、错误分类、错误定位与修正。我们以56,287条自然语言-形式语言对训练了诊断模型 FormalRx-8B,并发布首个细粒度诊断基准 FormalRx-Test。FormalRx-8B 在测试中取得0.88(判断)与0.71(分类)的F1分数,以及0.75(定位)与0.73(修正)的准确率,显著优于通用大模型与专用基线。通过连接评估与可行动见解,FormalRx 支持系统性诊断与改进自动形式化系统。
原文摘要 · Abstract (English)
The veracious semantic alignment in autoformalization is significant for formal mathematical reasoning. However, existing evaluations provide only opaque binary verdicts or scalar scores, offering no interpretable insight into where or why translations fail. This opacity severely limits both human understanding and automated system improvement. To bridge this gap, we introduce FormalRx, a comprehensive diagnostic evaluation framework that transforms autoformalization assessment from black-box judgments into actionable feedback. At its core is SCI Error Taxonomy, a hierarchical classification scheme decomposing autoformalization errors into 28 distinct categories with strict priority ordering. Building on this taxonomy, FormalRx provides four critical diagnostic capabilities: alignment verdicts, error categorization, error localization, and correction. We instantiate the framework with a diagnostic model FormalRx-8B, trained on 56,287 NL-FL pairs with fine-grained diagnostic annotations, and release FormalRx-Test as the first fine-grained diagnostic benchmark. FormalRx-8B achieves F1-scores of 0.88 (verdict) and 0.71 (categorization), along with accuracies of 0.75 (localization) and 0.73 (correction), substantially outperforming both general-purpose LLMs and specialized baselines. By connecting evaluation with actionable insights, FormalRx enables systematic diagnosis and improvement of autoformalization systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。