小模型微调后答题更准，大模型却更适应新场景。
Question: How do Large Language Models perform on the Question Answering tasks? Answer:
- 用单次推理提示替代双次推理，节省计算资源。
- 微调小模型在原任务上优于大模型，但大模型在新数据上表现更好。
- 适合关注模型泛化能力与实际部署效率的研究者。
大型语言模型（LLMs）在无需显式训练的情况下，通过少量示例或零样本提示即可在多种自然语言处理任务中取得良好效果。问答（QA）是其中常见任务。本研究在SQuAD2数据集上，对比了微调的小模型与现成的指令遵循型大模型在单次推理提示下的表现。由于该数据集包含无法回答的问题，以往工作采用双次推理方法。本文提出一种新提示策略，在不依赖双次推理的前提下实现相似效果，节省计算成本。此外，我们通过在5个不同但相似的QA数据集上测试模型的泛化能力，未对任一模型进行微调，模拟真实场景中上下文和问题分布变化的情况。结果表明：微调小模型在原任务上仍优于当前最先进（SOTA）大模型；但最新大模型能缩小差距，在3个新数据集上甚至超越微调模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have been showing promising results for various NLP-tasks without the explicit need to be trained for these tasks by using few-shot or zero-shot prompting techniques. A common NLP-task is question-answering (QA). In this study, we propose a comprehensive performance comparison between smaller fine-tuned models and out-of-the-box instruction-following LLMs on the Stanford Question Answering Dataset 2.0 (SQuAD2), specifically when using a single-inference prompting technique. Since the dataset contains unanswerable questions, previous work used a double inference method. We propose a prompting style which aims to elicit the same ability without the need for double inference, saving compute time and resources. Furthermore, we investigate their generalization capabilities by comparing their performance on similar but different QA datasets, without fine-tuning neither model, emulating real-world uses where the context and questions asked may differ from the original training distribution, for example swapping Wikipedia for news articles. Our results show that smaller, fine-tuned models outperform current State-Of-The-Art (SOTA) LLMs on the fine-tuned task, but recent SOTA models are able to close this gap on the out-of-distribution test and even outperform the fine-tuned models on 3 of the 5 tested QA datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。