用图像代替代码文本,实现高达8倍的压缩,提升代码理解效率。
Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding
- 将代码转为图像输入,利用视觉模型实现高效压缩
- 4倍压缩下仍保持良好代码补全性能,8倍压缩不影响克隆检测
- 适合追求推理效率的开发者与大型代码系统优化场景
大语言模型在源码理解中表现优异,但随着软件规模扩大,计算效率成为瓶颈。现有模型依赖文本序列处理方式,导致上下文长度和计算成本线性增长。多模态大模型(MLLMs)的发展提供了新思路:将源码渲染为图像进行表示。相比难以压缩的文本,图像可通过调整分辨率大幅降低令牌开销,同时保持语义可读性。我们首次系统研究了MLLM在代码理解中的有效性。实验表明:(1)MLLM可在代码理解任务中实现最高8倍的令牌压缩;(2)在4倍压缩下,能有效利用语法高亮等视觉线索,提升代码补全性能;(3)克隆检测等任务对视觉压缩具有极强鲁棒性,部分压缩比甚至优于原始文本输入。结果揭示了多模态代码表示在提升推理效率方面的潜力与局限。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original token cost while remaining recognizable to vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。