用少量示例让大模型逻辑分解政策,提升对话问答的可解释性。
Few-shot Policy (de)composition in Conversational Question Answering
- 基于提示工程与逻辑分解,仅需少量样例实现政策合规判断
- 在ShARC数据集上表现媲美微调模型,无需任务特定训练
- 生成显式逻辑图,可定位错误并揭示数据集歧义
政策合规检测(PCD)旨在判断场景是否符合既定政策。在对话场景中,其结果可指示是否需要提出澄清问题以确定合规状态。现有方法通常声称具备隐含推理能力或依赖大量标注数据。本文提出逻辑分解政策合规(LDPC):一种基于大语言模型的神经符号框架,在少样本设置下检测政策合规性。通过选择少量示例并结合最新提示技术,该方法能有效提取待回答的子问题,从上下文信息中分配真值,并从给定政策显式生成逻辑语句。显式逻辑图的构建提升了对相关问题的透明度与可解释性。我们将该方法应用于主流的PCD与对话机器阅读基准ShARC,取得与微调模型相当的性能,且无需任务特定微调。同时,利用LDPC固有的可解释架构,我们分析了错误发生位置,揭示了ShARC数据集中的歧义,并凸显了对话问答中推理的挑战。
原文摘要 · Abstract (English)
The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of PCD can indicate if clarifying questions must be asked to determine compliance status. Existing approaches usually claim to have reasoning capabilities that are latent or require a large amount of annotated data. In this work, we propose logical decomposition for policy compliance (LDPC): a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting. By selecting only a few exemplars alongside recently developed prompting techniques, we demonstrate that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies. The formulation of explicit logic graphs can in turn help answer PCDrelated questions with increased transparency and explainability. We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning. We also leverage the inherently interpretable architecture of LDPC to understand where errors occur, revealing ambiguities in the ShARC dataset and highlighting the challenges involved with reasoning for conversational question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。