轻量级基准测试工具，快速对比不同检索模型与效率设置的性能表现。
HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions
- 将多个检索任务压缩为小规模统一数据集，支持跨模型公平比较。
- 覆盖43种语言、551个任务，与主流基准相关性超过0.97。
- 适合研发人员快速筛选模型、检测回归问题或探索性能-效率权衡。
随着检索增强生成和语义搜索的快速发展，选择合适的嵌入方法与检索配置变得愈发困难。现有大型基准虽全面但过于沉重，难以在开发中频繁复现，且缺乏对维度压缩、量化、重排序等生产级优化设置进行统一条件比较的基础设施。本文提出 HAKARI-Bench，一个轻量级基准，将现有检索套件重构为小型数据集（Nano-sets）：包含35个基准、551个任务，覆盖43种语言，采用统一格式。该基准支持在相同条件下，对五类检索架构（BM25、密集型、稀疏型、晚期交互、重排序器）及其效率变体进行模型无关的比较。在55个模型上，其整体排名与官方MTEB retrieval v2、MMTEB v2 retrieval及English BEIR（完整版）的相关性均超过Spearman 0.97。HAKARI-Bench不替代全量评估，而是支持快速模型选型、回归检测与质量-效率帕累托前沿分析。代码、数据与排行榜已开源，许可协议为MIT。
原文摘要 · Abstract (English)
With the rapid spread of retrieval-augmented generation and semantic search, choosing the right embedding and retrieval configuration is increasingly hard. Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings--dimensionality reduction, quantization, reranking--across many models under identical conditions. We present HAKARI-Bench, a lightweight benchmark that reconstructs existing retrieval suites into small datasets (Nano-sets): 35 benchmarks and 551 tasks across 43 languages in a unified format, enabling same-condition, model-agnostic comparison of five retrieval families (BM25, dense, sparse, late interaction, rerankers) and their efficiency variants. Across 55 models, its overall ranking reproduces the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) at Spearman >0.97. HAKARI-Bench does not replace full evaluation; it enables rapid model selection, regression detection, and reading the quality-efficiency Pareto frontier. Code, data, and leaderboard are released under the MIT license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。