一站式评测工具，高效评估大模型多语言多模态能力。
OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions
- 模块化设计支持快速集成新模型与数据集。
- 覆盖100+模型、50+数据集，支持数千种组合评测。
- 轻量快速部署，适合研究者与开发者日常评测使用。
大型语言模型（LLMs）的快速发展使其应用范围扩展至多语言、领域专用任务及多模态融合。本文提出OmniEvalKit，一个新型基准评测工具箱，用于评估LLMs及其全场景扩展在多语言、多领域和多模态能力上的表现。不同于仅聚焦单一维度的现有基准，OmniEvalKit提供模块化、轻量化且自动化的评估系统，采用静态构建器与动态数据流的架构，支持新模型与数据集的无缝接入。该工具箱支持超过100个LLM和50个评估数据集，覆盖数千种模型-数据集组合的全面评测。其设计目标是打造超轻量、快速部署的评估框架，为人工智能社区的下游应用提供更便捷、灵活的评测支持。
原文摘要 · Abstract (English)
The rapid advancements in Large Language Models (LLMs) have significantly expanded their applications, ranging from multilingual support to domain-specific tasks and multimodal integration. In this paper, we present OmniEvalKit, a novel benchmarking toolbox designed to evaluate LLMs and their omni-extensions across multilingual, multidomain, and multimodal capabilities. Unlike existing benchmarks that often focus on a single aspect, OmniEvalKit provides a modular, lightweight, and automated evaluation system. It is structured with a modular architecture comprising a Static Builder and Dynamic Data Flow, promoting the seamless integration of new models and datasets. OmniEvalKit supports over 100 LLMs and 50 evaluation datasets, covering comprehensive evaluations across thousands of model-dataset combinations. OmniEvalKit is dedicated to creating an ultra-lightweight and fast-deployable evaluation framework, making downstream applications more convenient and versatile for the AI community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。