研究量化与蒸馏如何影响代码令牌的表示，揭示压缩模型中的隐藏机制。
Compressed code: the hidden effects of quantization and distillation on programming tokens
- 提出冷启动概率分析法，无需提示即可洞察模型行为。
- 发现量化和蒸馏会显著改变代码令牌分布，影响生成质量。
- 适合关注模型压缩与代码生成优化的工程师和研究者。
大型语言模型（LLMs）在代码生成方面表现卓越，但其令牌级机制仍不明确，尤其在压缩模型中。通过系统分析编程语言令牌表征，我们刻画了编程语言在LLM分词器中的编码方式，分析其词汇分布与关键词覆盖模式。引入一种新型冷启动概率分析方法，可在无需显式提示的情况下提供对模型行为的洞见。此外，全面评估了量化、蒸馏、模型缩放及任务特定微调等不同模型优化技术对令牌级表征和代码生成质量的影响。实验基于全面的概率分布分析与评估指标，揭示了令牌级行为的关键洞察，并提供了在各种优化约束下维持代码生成质量的实证指导。这些发现推动了对LLM代码生成的理论理解，并为生产环境中优化模型的实践提供了支持。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。