用强化学习训练大模型做自然语言推理，无需标注理由。
Pushing the boundary on Natural Language Inference
- 用GRPO强化学习让模型自动生成推理链，不依赖人工标注。
- 32B量化模型在11个对抗性数据集上7个超顶尖水平，仅需22GB显存。
- 适合想构建高效可靠推理系统的研究者与工程师。
自然语言推理（NLI）是自然语言理解的核心任务，广泛应用于事实核查、问答和信息检索。尽管重要，现有NLI系统严重依赖带偏差的标注数据进行监督学习，限制了泛化能力与实际应用。本文提出基于组相对策略优化（GRPO）的强化学习方法，用于链式思维（CoT）学习，无需标注推理过程，可在更具挑战性的ANLI等数据集上训练。我们使用参数高效微调技术（LoRA与QLoRA）对7B、14B和32B语言模型进行微调，在标准与对抗性NLI基准上表现优异。其中32B AWQ量化模型在11个对抗性子集中有7个超越当前最佳结果，若考虑复现则全部领先，且仅需22GB内存占用，证明强推理能力可在激进量化下保留。本工作为构建鲁棒、可扩展且实用的NLI系统提供了有效框架。
原文摘要 · Abstract (English)
Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, and information retrieval. Despite its importance, current NLI systems heavily rely on supervised learning with datasets that often contain annotation artifacts and biases, limiting generalization and real-world applicability. In this work, we apply a reinforcement learning-based approach using Group Relative Policy Optimization (GRPO) for Chain-of-Thought (CoT) learning in NLI, eliminating the need for labeled rationales and enabling this type of training on more challenging datasets such as ANLI. We fine-tune 7B, 14B, and 32B language models using parameter-efficient techniques (LoRA and QLoRA), demonstrating strong performance across standard and adversarial NLI benchmarks. Our 32B AWQ-quantized model surpasses state-of-the-art results on 7 out of 11 adversarial sets$\unicode{x2013}$or on all of them considering our replication$\unicode{x2013}$within a 22GB memory footprint, showing that robust reasoning can be retained under aggressive quantization. This work provides a scalable and practical framework for building robust NLI systems without sacrificing inference quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。