首次评估压缩版大模型的智能体能力，发现4比特量化影响实际应用表现。
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
- 构建ACBench基准，涵盖任务流程、工具调用等四大智能体能力
- 4比特量化使实际应用准确率下降10%-15%，但任务生成能力仅降1%-3%
- 提出新评估指标，助力优化智能体场景下的模型压缩
后训练压缩可降低大语言模型（LLM）的计算与内存开销，实现资源高效部署。然而现有压缩评估仅关注语言建模（如困惑度）和自然语言理解任务（如GLUE准确率），忽视了智能体能力——包括工作流生成、工具调用、长上下文理解及真实世界应用。本文提出首个综合性评估基准ACBench，覆盖12项任务、4类能力（如WorfBench用于工作流生成，Needle-in-Haystack用于长上下文检索）、两种压缩方法（量化GPTQ/AWQ，剪枝Wanda/SparseGPT）及15个模型（含小型模型Gemma-2B、标准模型Qwen2.5 7B-32B及蒸馏推理模型DeepSeek-R1-Distill）。实验揭示压缩权衡：4比特量化对工作流生成与工具调用影响极小（下降1%-3%），但真实应用准确率下降10%-15%。本文引入ERank、Top-k排名相关性与能量指标，系统化分析压缩影响。ACBench为智能体场景下模型压缩优化提供可操作洞见。代码已开源：https://github.com/pprp/ACBench。
原文摘要 · Abstract (English)
Post-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmarks only focus on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities - workflow, tool use/function call, long-context understanding and real-world application. We introduce the Agent Compression Benchmark (ACBench), the first comprehensive benchmark for evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1) 12 tasks across 4 capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ) and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B), standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill). Our experiments reveal compression tradeoffs: 4-bit quantization preserves workflow generation and tool use (1%-3% drop) but degrades real-world application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation and Energy to systematize analysis. ACBench provides actionable insights for optimizing LLM compression in agentic scenarios. The code can be found in https://github.com/pprp/ACBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。