对比47个问答模型在8个数据集的表现,找出现成可用的最佳模型。
Comparative Analysis of 47 Context-Based Question Answer Models Across 8 Diverse Datasets
- 在不微调情况下测试47个Hugging Face模型的性能
- Electra-large模型在多数数据集上表现最佳,平均准确率43%
- 适合快速部署、无需定制训练的问答系统应用
基于上下文的问答(CBQA)模型通过利用上下文信息提供更准确和相关答案,在用户支持、信息检索和教育平台等场景中具有广泛应用。本文在8个不同数据集上基准测试了来自Hugging Face的47个CBQA模型,旨在识别无需额外微调即可在多种数据集上表现最优的模型。结果表明,最佳模型为ahotrod/electra_large_discriminator_squad2_512,整体平均准确率为43%。该模型在bioasq10b-factoid(65.92%)、biomedical_cpgQA(96.45%)、QuAC(11.13%)和Question Answer Dataset(41.6%)上表现突出。Bert-large-uncased-whole-word-masking-finetuned-squad在IELTS数据集上达到82%准确率。模型计算时间与上下文长度和模型规模正相关,且随着答案长度增加,性能通常下降;此外,上下文复杂度也影响模型表现。我们还使用遗传算法融合多个模型输出以提升整体准确率。
原文摘要 · Abstract (English)
Context-based question answering (CBQA) models provide more accurate and relevant answers by considering the contextual information. They effectively extract specific information given a context, making them functional in various applications involving user support, information retrieval, and educational platforms. In this manuscript, we benchmarked the performance of 47 CBQA models from Hugging Face on eight different datasets. This study aims to identify the best-performing model across diverse datasets without additional fine-tuning. It is valuable for practical applications where the need to retrain models for specific datasets is minimized, streamlining the implementation of these models in various contexts. The best-performing models were trained on the SQuAD v2 or SQuAD v1 datasets. The best-performing model was ahotrod/electra_large_discriminator_squad2_512, which yielded 43\% accuracy across all datasets. We observed that the computation time of all models depends on the context length and the model size. The model's performance usually decreases with an increase in the answer length. Moreover, the model's performance depends on the context complexity. We also used the Genetic algorithm to improve the overall accuracy by integrating responses from other models. ahotrod/electra_large_discriminator_squad2_512 generated the best results for bioasq10b-factoid (65.92\%), biomedical\_cpgQA (96.45\%), QuAC (11.13\%), and Question Answer Dataset (41.6\%). Bert-large-uncased-whole-word-masking-finetuned-squad achieved an accuracy of 82\% on the IELTS dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。