arXiv:2505.11441cs.CL2025-05被引 2

发现代码压缩与智能非线性相关,呈对数关系。

Is Compression Really Linear with Code Intelligence?

  • 提出格式淬炼方法,公平评估代码大模型能力
  • 实测显示压缩率(BPC)与代码智能呈对数关系
  • 适合关注代码生成、模型评估的研究者

理解数据压缩与大型语言模型(LLM)能力之间的关系至关重要,尤其在代码智能等专业领域。以往研究认为压缩与通用智能呈线性关系,但忽略了代码涵盖多种编程语言和任务的复杂性,且难以公正评估现代代码大模型。本文通过在多语言、多任务代码基准上评估一系列开源代码大模型,解决预训练模型代码智能的高效公平评估难题。我们提出轻量级、透明的训练方法「格式淬炼」,以评估模型内在能力。使用来自GitHub的全新大规模代码验证集,以比特/字符(BPC)度量压缩效果。实证结果揭示代码智能与BPC之间存在根本性的对数关系。这修正了先前的线性假设,我们认为线性关系可能是特定有限条件下对数曲线尾部的误判。本工作深化了对压缩在代码智能中作用的理解,并为代码领域提供了稳健的评估框架。

原文摘要 · Abstract (English)

Understanding the relationship between data compression and the capabilities of Large Language Models (LLMs) is crucial, especially in specialized domains like code intelligence. Prior work posited a linear relationship between compression and general intelligence. However, it overlooked the multifaceted nature of code that encompasses diverse programming languages and tasks, and struggled with fair evaluation of modern Code LLMs. We address this by evaluating a diverse array of open-source Code LLMs on comprehensive multi-language, multi-task code benchmarks. To address the challenge of efficient and fair evaluation of pre-trained LLMs' code intelligence, we introduce \textit{Format Annealing}, a lightweight, transparent training methodology designed to assess the intrinsic capabilities of these pre-trained models equitably. Compression efficacy, measured as bits-per-character (BPC), is determined using a novel, large-scale, and previously unseen code validation set derived from GitHub. Our empirical results reveal a fundamental logarithmic relationship between measured code intelligence and BPC. This finding refines prior hypotheses of linearity, which we suggest are likely observations of the logarithmic curve's tail under specific, limited conditions. Our work provides a more nuanced understanding of compression's role in developing code intelligence and contributes a robust evaluation framework in the code domain.

代码智能大模型评估压缩率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。