用动态埃洛评分评估大模型在社科文本分类中的表现
TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences
- 基于定制埃洛系统，持续评估大模型在社科文本分类能力
- 首周期测试中四语言（中英德俄）不文明言论分类表现各异
- 支持多语言、多任务扩展，适合关注模型泛化能力的研究者
TextClass基准项目是一个持续进行的评估流程，旨在全面、公平且动态地评测大语言模型和变换器在文本分类任务中的表现。评估覆盖社会科学领域中使用自然语言处理与文本作为数据方法的多种学科和语言。排行榜采用定制的埃洛评分系统展示性能指标与相对排名。每轮榜单更新时，新模型加入，固定测试集可替换为未见等效数据以检验泛化能力，评分随之调整，并生成综合加权的元埃洛排行榜。本文阐述项目动机，详述埃洛评分机制，并估算不同社科分类任务下的元埃洛值。同时呈现首个周期在中文、英文、德语和俄语不文明言论数据上的分类结果。该持续评估体系还涵盖阿拉伯语、印地语、西班牙语等更多语言，以及政策议程主题、虚假信息等分类任务。
原文摘要 · Abstract (English)
The TextClass Benchmark project is an ongoing, continuous benchmarking process that aims to provide a comprehensive, fair, and dynamic evaluation of LLMs and transformers for text classification tasks. This evaluation spans various domains and languages in social sciences disciplines engaged in NLP and text-as-data approach. The leaderboards present performance metrics and relative ranking using a tailored Elo rating system. With each leaderboard cycle, novel models are added, fixed test sets can be replaced for unseen, equivalent data to test generalisation power, ratings are updated, and a Meta-Elo leaderboard combines and weights domain-specific leaderboards. This article presents the rationale and motivation behind the project, explains the Elo rating system in detail, and estimates Meta-Elo across different classification tasks in social science disciplines. We also present a snapshot of the first cycle of classification tasks on incivility data in Chinese, English, German and Russian. This ongoing benchmarking process includes not only additional languages such as Arabic, Hindi, and Spanish but also a classification of policy agenda topics, misinformation, among others.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。