大模型能否替代微调模型？关键看意图空间大小。
When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems
- 对比微调RoBERTa与零样本大模型在不同场景下的表现
- 小意图集时微调模型更优，大意图集时两者相当
- 适合快速部署、抗噪声和动态更新的生产系统
有观点认为零样本大语言模型可替代微调的NLU分类器进行意图识别。我们通过实证比较发现：这取决于意图空间规模。在全量ATIS和CLINC150数据集上，对比了微调RoBERTa、TF-IDF+逻辑回归基线、句子嵌入kNN以及Claude Haiku零样本模型，采用自举95%置信区间与配对显著性检验。当存在充足领域标注数据时，微调RoBERTa性能更优且成本低三个数量级：在ATIS上比Claude零样本高11.8分（95.9 vs. 84.1，p<0.001）。在包含150个意图的广泛CLINC150任务中，两者无统计差异（89.1 vs. 88.5，p=0.24）——大模型无需训练即可媲美全监督模型。大模型优势体现在三类生产场景：泛化出域检测（OOS召回率85.6 vs. RoBERTa的58.1）、对真实语音识别噪声的鲁棒性（在0dB下达到92.5 vs. 80.0），以及动态部署模式下，同一模型适配多个应用意图，准确率达~94%而无需重训练。本文据此提出实用决策框架。
原文摘要 · Abstract (English)
A common claim is that zero-shot large language models (LLMs) can replace fine-tuned NLU classifiers for intent detection. We test this claim head-to-head and find that the honest answer is: it depends on the intent space. On full ATIS and CLINC150 we compare a fine-tuned RoBERTa, a TF-IDF+logistic-regression baseline, sentence-embedding kNN, and Claude Haiku zero-shot, reporting bootstrap 95% confidence intervals and paired significance tests. When abundant in-domain labels exist, fine-tuned RoBERTa is as good or better and three orders of magnitude cheaper and faster: on ATIS it beats Claude zero-shot by 11.8 points (95.9 vs. 84.1, p<0.001). On the broad 150-intent CLINC150 schema the two are statistically tied (89.1 vs. 88.5, p=0.24): the LLM matches a fully supervised model with no training data. The LLM's advantages appear in three production-relevant regimes: out-of-scope detection (OOS recall 85.6 vs. 58.1 for RoBERTa); robustness to realistic ASR noise via a controlled text-to-speech to noise to Whisper pipeline (92.5 vs. 80.0 at 0 dB); and dynamic per-deployment schemas, where a classifier trained on one app's intents scores 0% on a new app's intents while the schema-prompted LLM serves both at ~94% with zero retraining. We distill these findings into a decision framework for practitioners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。