提出新方法识别文本模糊性,提升大模型在工具调用中的准确性
Ambiguity in LLMs is a concept missing problem
- 基于概念路径核度量文本模糊性,捕捉语义差异
- 在工具调用任务中实现最优性能,超越现有方法
- 适合需要高精度文本转结构化数据的场景
自然语言中的模糊性是大语言模型(LLMs)实现文本到结构化数据映射时的重要障碍,影响文本转工具调用和文本转SQL等任务。现有方法依赖ReACT框架通过试错获得正确映射,或通过监督微调使模型偏向特定任务。本文提出新思路:在潜在空间表征中刻画模糊文本的差异,并利用这些差异在映射前识别模糊性。为检测句级模糊性,关注模糊问题与其解释之间的关系。不同于密集嵌入的距离度量,引入基于概念路径核的新距离度量,识别出可区分模糊与非模糊问题的模式。此外,提出通过缺失概念预测来提升大模型在模糊工具调用任务上的表现,均达到当前最佳效果。
原文摘要 · Abstract (English)
Ambiguity in natural language is a significant obstacle for achieving accurate text to structured data mapping through large language models (LLMs), which affects the performance of tasks such as mapping text to agentic tool calling and text-to-SQL queries. Existing methods to ambiguity handling either rely on the ReACT framework to obtain correct mappings through trial and error, or on supervised fine-tuning to bias models toward specific tasks. In this paper, we adopt a different approach that characterizes representation differences of ambiguous text in the latent space and leverages these differences to identify ambiguity before mapping them to structured data. To detect sentence-level ambiguity, we focus on the relationship between ambiguous questions and their interpretations. Unlike distances calculated by dense embeddings, we introduce a new distance measure based on a path kernel over concepts. With this measurement, we identify patterns to distinguish ambiguous from unambiguous questions. Furthermore, we propose a method for improving LLM performance on ambiguous agentic tool calling through missing concept prediction. Both achieve state-of-the-art results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。