用符号逻辑验证大模型是否真听懂指令，避免错误传播。
Neuro-Symbolic Verification on Instruction Following of LLMs
- 将指令转为逻辑与语义约束，统一求解验证
- 在新基准上准确率显著优于纯大模型方法
- 可解释反馈助力模型改进，无需额外训练
大语言模型在关键应用中常不遵循指令，且错误难以察觉。在基于大模型的智能体工作流中，此类错误会沿推理链传播并放大，导致任务失败。本文提出NSVIF，一种无假设、通用的神经符号验证框架，用于检验大模型输出是否遵循提示指令。该框架将指令遵循问题建模为约束满足问题，同时处理逻辑与语义约束，并由统一求解器协同执行逻辑推理与语义分析。为评估性能，构建了带有细粒度标签的新基准VIFBENCH。实验表明，NSVIF显著优于基于大模型的验证方法，且能提供可解释反馈；该反馈还能在不进行后训练的前提下提升大模型的指令遵循能力。
原文摘要 · Abstract (English)
A fundamental problem of applying Large Language Models (LLMs) to important applications is that LLMs do not always follow instructions, and violations are often hard to observe or check. In LLM-based agentic workflows, such violations can propagate and amplify along reasoning chains, causing task failures and system incidents. This paper presents NSVIF, a neuro-symbolic framework for verifying whether an LLM's output follows the instructions used to prompt the LLM. NSVIF is a universal, general-purpose verifier; it makes no assumption about the instruction or the LLM. NSVIF formulates instruction-following verification as a constraint-satisfaction problem by modeling user instructions as constraints. NSVIF models both logical and semantic constraints; constraint solving is done by a unified solver that orchestrates logical reasoning and semantic analysis. To evaluate NSVIF, we develop VIFBENCH, a new benchmark for instruction-following verifiers with fine-grained data labels. Experiments show that NSVIF significantly outperforms LLM-based approaches and provides interpretable feedback. We also show that feedback from NSVIF helps improve LLMs' instruction-following capability without post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。