用合成语义描述训练小模型,提升代码表征效果。
Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study

- 用合成自然语言描述代码功能,构建对比学习信号
- 在8个任务中5项超越同类小模型,2项持平
- 适合资源有限但需高效代码表征的场景
通用代码嵌入支持代码搜索、分类和检索等工具。小型Transformer编码器通常依赖人工编写的文档字符串(耗时且不一致)或挖掘的执行痕迹(特定场景且收集成本高)。本文实证研究一种替代方案:在双编码器框架中,用合成生成的自然语言描述(强调代码功能与意图)作为对比学习的监督信号,训练小型编码器,该描述仅用于训练,推理时丢弃。在C、C++和Java的8个检索、分类和生成任务上,与基于预训练的基线、通用大模型及专用嵌入模型进行对比。结果显示,合成语义监督在5个任务上显著优于同尺寸预训练基线,在另外2个任务上达到相当水平;微调后,其分类性能匹配或超过大2个数量级的零样本模型,且在相同预训练数据下与执行感知监督表现相当,表明这是一种可扩展、高效的代码表征新范式。
原文摘要 · Abstract (English)
General-purpose code embeddings power tools for code search, classification, and retrieval. Compact transformer encoders for code typically rely on either human-written docstrings (labor-intensive and inconsistent) or mined structural signals such as execution traces (setting-specific and costly to collect). We empirically study an alternative: contrastive pretraining of small encoders with synthetically generated natural-language descriptions emphasizing code functionality and intent, paired with code in a dual-encoder framework at training and discarded at inference. We benchmark this approach against pretraining-based baselines, generalist LLMs, and embedding-specific models on eight retrieval, classification, and generation tasks across C, C++, and Java. Synthetic semantic supervision yields statistically significant gains over pretraining baselines of the same inference-time size on five of eight tasks, with parity on two more; once fine-tuned, it matches or exceeds zero-shot models two orders of magnitude larger on classification, and it stays on par with execution-aware supervision at matched pretraining data, suggesting a scalable, effective alternative to existing code-representation paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。