通过分层一致性蒸馏，提升自然语言转逻辑公式的准确率。
Stratified Consistency Distillation for Natural Language Formalization
- 对同一输入生成多个逻辑翻译并按语义聚类，分层处理不确定性。
- 在Pass@K和新提出的等价逻辑相似度上均显著提升。
- 适合需要高精度逻辑转换的复杂推理任务研究者。
神经符号推理通过结合大语言模型（LLMs）与符号求解器，在解决复杂推理任务方面展现出良好前景。然而，一个根本性挑战依然存在：如何提高自然语言到逻辑公式翻译的准确性。现有方法主要依赖提示工程，难以在不同领域和输入格式间扩展。受其他模型微调与对齐应用成功的启发，我们提出一种基于微调的分层一致性蒸馏方法：(1) 使用前沿大模型为每个输入生成 K 个逻辑翻译，并按语义等价性进行聚类；(2) 根据熵值水平，分别采用多数投票（低熵）、大模型作为裁判（中熵）或统一/放弃（高熵）策略；(3) 使用选定的伪标签微调小型模型。实验表明，该方法在 Pass@K 和我们提出的新型等价逻辑相似度指标上均实现显著且一致的提升，证明了通过一致性蒸馏推进逻辑翻译的潜力。
原文摘要 · Abstract (English)
Neurosymbolic reasoning has shown promising success in addressing complex reasoning tasks by combining large language models (LLMs) and symbolic solvers. While this approach shows promise, a fundamental challenge remains: improving the accuracy of translations from natural language to logical formulas. Current methods predominantly rely on prompt engineering, which is difficult to scale across different domains and input formats. Drawing inspiration from the success of fine-tuning in other model adaptation and alignment applications, we propose a fine-tuning-based Stratified Consistency Distillation approach: (1) We generate K logical translations per input using a frontier LLM and cluster them by semantic equivalence (2) Based on the entropy level, we apply majority voting (low entropy), LLM-as-a-Judge (medium entropy), or unification/abstention (high entropy), and (3) fine-tune a smaller model using the selected pseudo-labels. Our experiments show significant and consistent improvements in both Pass@K and our novel Equivalent Logical Similarity metrics, demonstrating the potential of advancing logical translation through consistency distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。