模型热度不等于性能，80%文档不全，88%夸大效果。
Model Hubs and Beyond: Analyzing Model Popularity, Performance, and Documentation
- 评估500个情感分析模型，人工标注近8万条数据。
- 热门模型性能不优，88%作者夸大性能表现。
- 提供选型指南，帮助用户避开虚假宣传模型。
随着机器学习模型在Hugging Face等平台激增，用户常因模型下载量、点赞数或更新时间等热度指标难以抉择。本文系统评估了平台上500个情感分析模型的性能与文档质量。研究通过近80,000次人工标注、大量模型训练与评测，发现模型热度与实际性能无强相关性。约80%的模型缺乏对模型结构、训练过程及评估方法的详细说明；约88%的作者在模型卡片中夸大其性能。基于此，本文提出一套选型检查清单，助力用户更科学地选择下游任务适用模型。
原文摘要 · Abstract (English)
With the massive surge in ML models on platforms like Hugging Face, users often lose track and struggle to choose the best model for their downstream tasks, frequently relying on model popularity indicated by download counts, likes, or recency. We investigate whether this popularity aligns with actual model performance and how the comprehensiveness of model documentation correlates with both popularity and performance. In our study, we evaluated a comprehensive set of 500 Sentiment Analysis models on Hugging Face. This evaluation involved massive annotation efforts, with human annotators completing nearly 80,000 annotations, alongside extensive model training and evaluation. Our findings reveal that model popularity does not necessarily correlate with performance. Additionally, we identify critical inconsistencies in model card reporting: approximately 80% of the models analyzed lack detailed information about the model, training, and evaluation processes. Furthermore, about 88% of model authors overstate their models' performance in the model cards. Based on our findings, we provide a checklist of guidelines for users to choose good models for downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。