用专用语言让大模型写逻辑代码，再交由求解器执行，准确率超90%。
Logic.py: Bridging the Gap between LLMs and Constraint Solvers
- 让大模型用Logic.py语言描述逻辑题，再交给约束求解器计算。
- 在ZebraLogicBench上准确率达90%以上，比基线提升65个百分点。
- 适合想用大模型解决复杂逻辑问题的研究者和开发者。
我们提出一种新方法，利用大语言模型形式化并求解搜索类问题，显著优于此前的最先进水平。该方法在逻辑谜题基准测试ZebraLogicBench上得到验证。与直接让大模型求解不同，我们的方法引导模型将问题用一种面向逻辑的领域特定语言（DSL）Logic.py进行形式化表达，随后由约束求解器完成求解，融合了语言模型与求解器的优势。该方法在Llama 3.1 70B基线上实现了65%的绝对性能提升，在ZebraLogicBench上达到超过90%的准确率，创下新纪录。这一显著进展展示了将大语言模型与领域特定语言及外部工具结合，在传统上对大模型具有挑战的任务中的巨大潜力。
原文摘要 · Abstract (English)
We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。