构建首个全面评估大模型意图理解能力的基准，发现现有模型表现远低于人类。
IntentGrasp: A Comprehensive Benchmark for Intent Understanding
- 从49个数据集整合12个领域，统一任务格式构建大规模意图理解数据集。
- 20个大模型在测试集上准确率不足60%，部分甚至低于随机猜测水平。
- 提出针对性微调方法IFT，显著提升模型在跨领域场景下的意图理解能力。
准确理解语音、对话和文本背后的意图，对发展有用的大语言模型助手至关重要。本文提出IntentGrasp，一个用于评估大模型意图理解能力的综合性基准。该基准源自49个高质量、开源许可的数据集，覆盖12个不同领域，通过数据集筛选、意图标签上下文化与任务格式统一构建而成。IntentGrasp包含一个262,759条样本的大规模训练集，以及两个评测集：包含12,909个测试用例的All Set，和更具平衡性与挑战性的470个Gem Set。对20个来自7个家族的大模型（包括GPT-5.4、Gemini-3.1-Pro、Claude-Opus-4.7等前沿模型）的广泛评估显示，其性能令人失望，在All Set上得分低于60%，在Gem Set上低于25%。值得注意的是，20个模型中有17个在Gem Set上的表现劣于随机基线（15.2%），而人类估计表现约为81.1%，表明仍有巨大提升空间。为增强该能力，本文提出意图微调（Intentional Fine-Tuning, IFT），在IntentGrasp训练集上对模型进行微调，使模型在All Set上F1分数提升30+点，在Gem Set上提升20+点。留一域排除（Lodo）实验进一步验证了IFT具有强跨域泛化能力，证实其是显著提升大模型意图理解能力的可行路径。总体而言，本研究通过基准评测与能力增强，为打造更具备意图理解能力、更可靠、更安全的智能助手提供了重要方向。
原文摘要 · Abstract (English)
Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assistants. This paper introduces IntentGrasp, a comprehensive benchmark for evaluating the intent understanding capability of LLMs. Derived from 49 high-quality, open-licensed corpora spanning 12 diverse domains, IntentGrasp is constructed through source datasets curation, intent label contextualization, and task format unification. IntentGrasp contains a large-scale training set of 262,759 instances and two evaluation sets: an All Set of 12,909 test cases and a more balanced and challenging Gem Set of 470 cases. Extensive evaluations on 20 LLMs across 7 families (including frontier models such as GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.7) demonstrate unsatisfactory performance, with scores below 60% on All Set and below 25% on Gem set. Notably, 17 out of 20 tested models perform worse than a random-guess baseline (15.2%) on Gem Set, while the estimated human performance is ~81.1%, showing substantial room for improvement. To enhance such ability, this paper proposes Intentional Fine-Tuning (IFT), which fine-tunes the models on the training set in IntentGrasp, yielding significant gains of 30+ F1 points on All Set and 20+ points on Gem Set. Tellingly, the leave-one-domain-out (Lodo) experiments further demonstrate the strong cross-domain generalizability of IFT, verifying that it is a promising approach to substantially enhancing the intent understanding of LLMs. Overall, by benchmarking and boosting intent understanding ability, this study sheds light on a promising path towards more intentional, capable, and safe AI assistants for human benefits and social good.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。