用偏好优化提升大模型将自然语言逻辑题转为一阶逻辑的能力
LOGICPO: Efficient Translation of NL-based Logical Problems to FOL using LLMs and Preference Optimization
- 构建新数据集LogicPO，结合偏好优化微调开源模型
- Phi-3.5比GPT-3.5-turbo少14%语法错误，多10%逻辑正确
- 适合想提升模型逻辑推理能力的研究者与开发者
逻辑推理是人工智能的关键任务，对问答、摘要等下游任务至关重要。当前大模型在将自然语言推理问题准确转换为逻辑表达式方面仍存在不足，限制了整体推理能力。为此，我们提出使用偏好优化数据集进行微调，使模型能整体解析自然语言问题并生成一致的逻辑程序：1）构建新的监督与偏好优化数据集LogicPO；2）采用DPO、KTO等主流技术微调开源大模型。最佳模型Phi-3.5在8次提示下，逻辑正确率比GPT-3.5-turbo高10%，语法错误减少14%。通过该框架和改进的评估指标，为提升大模型逻辑推理能力提供了新方向。
原文摘要 · Abstract (English)
Logical reasoning is a key task for artificial intelligence due to it's role in major downstream tasks such as Question Answering, Summarization. Recent methods in improving the reasoning ability of LLMs fall short in correctly converting a natural language reasoning problem to an equivalent logical formulation, which hinders the framework's overall ability to reason. Towards this, we propose to use finetuning on a preference optimization dataset to learn to parse and represent a natural language problem as a whole to a consistent logical program by 1) introducing a new supervised and preference optimization dataset LogicPO, and 2) adopting popular techniques such as Direct Preference Optimization (DPO), Kahneman-Tversky optimization (KTO) to finetune open-source LLMs. Our best model with Phi-3.5 consistently outperforms GPT-3.5-turbo's (8-shot) by producing 10% more logically correct and with 14% less syntax errors. Through the framework and our improved evaluation metrics, we offer a promising direction in improving the logical reasoning of LLMs by better representing them in their logical formulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。