压缩长文本评测数据集，成本降95%仍保持高精度。
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
- 针对长文本评测冗余问题，提出针对性压缩方法。
- 仅237个样本，评估成本降至原版4.5%，相关性达0.97。
- 适合快速测试大模型长程理解能力的研究者使用。
长上下文理解（LCU）是当前大语言模型的重要研究方向。然而，由于长文本数据固有的长度特性，现有LCU评测基准常导致极高的评估成本，包括测试时间和推理开销。通过大量实验，我们发现现有LCU基准存在显著冗余，造成评估效率低下。本文提出一种针对稀疏信息长文本数据的简洁压缩方法，通过剪枝知名基准LongBench，构建MiniLongBench。该基准涵盖6个主要任务类别、21个具体任务，仅含237个测试样本。对超过60个LLM的实证分析表明，MiniLongBench将平均评估成本降至原版的4.5%，同时与LongBench结果的平均等级相关系数保持在0.97。因此，作为低成本基准，MiniLongBench有望大幅推动大模型长上下文理解能力的后续研究。
原文摘要 · Abstract (English)
Long Context Understanding (LCU) is a critical area for exploration in current large language models (LLMs). However, due to the inherently lengthy nature of long-text data, existing LCU benchmarks for LLMs often result in prohibitively high evaluation costs, like testing time and inference expenses. Through extensive experimentation, we discover that existing LCU benchmarks exhibit significant redundancy, which means the inefficiency in evaluation. In this paper, we propose a concise data compression method tailored for long-text data with sparse information characteristics. By pruning the well-known LCU benchmark LongBench, we create MiniLongBench. This benchmark includes only 237 test samples across six major task categories and 21 distinct tasks. Through empirical analysis of over 60 LLMs, MiniLongBench achieves an average evaluation cost reduced to only 4.5% of the original while maintaining an average rank correlation coefficient of 0.97 with LongBench results. Therefore, our MiniLongBench, as a low-cost benchmark, holds great potential to substantially drive future research into the LCU capabilities of LLMs. See https://github.com/MilkThink-Lab/MiniLongBench for our code, data and tutorial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。