arXiv:2608.26389cs.CLcs.LG2026-08

建立标准化平台,验证SVD压缩LLM的真实效果

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

论文配图:LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression
图 1 · 摘自论文原文
  • 统一评估基准与压缩比例,隔离低秩效应
  • 5种方法对比显示性能排名随模型和比例变化
  • 揭示精度下降与推理加速的矛盾,适合研究者复现

基于SVD的低秩压缩已成为降低大语言模型内存与计算成本的热门方向。然而,现有研究采用不同基准、压缩比和设置,难以公平比较,常混淆低秩方法本身与辅助技术的效果差异。因此,无法判断报告收益是方法改进还是评估协议导致。为解决此问题,我们提出LowRankArena——一个标准化的SVD压缩评估平台。该平台统一任务版本、统一精度压缩预算、统一比较范式与推理测量,并提供超过3 TiB的已压缩检查点。利用该平台,我们对五种代表性SVD方法进行统一审计发现:先前结论高度依赖特定协议,在不同模型与保持率下,性能领先者与层级均发生变化;多项选择准确率可能掩盖困惑度显著下降;名义上的低秩节省在实际工作负载中常带来有限甚至可忽略的端到端加速。代码已开源。

原文摘要 · Abstract (English)

SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.

模型压缩SVDLLM评估平台

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。