C2LLM用自适应注意力池化提升代码检索效果，7B模型在同类中排名第一。
C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling
- 用多头注意力池化生成代码嵌入，融合全部词元信息。
- 在300万数据上训练，7B模型在MTEB-Code榜单第一。
- 可灵活调整嵌入维度，适合需要高效代码检索的场景。
我们提出 C2LLM —— 对比代码大语言模型，包含 0.5B 与 7B 两种规模的代码嵌入模型。基于 Qwen-2.5-Coder 骨干网络，C2LLM 采用多头注意力池化（PMA）模块，从词元嵌入生成序列嵌入，有效利用预训练中获得的因果表示，同时聚合序列中所有词元的信息，突破传统以结束符（EOS）为基础的序列嵌入的信息瓶颈，并支持嵌入维度的灵活配置，可作为 MRL 的替代方案。在三百万公开可用数据上训练后，C2LLM 模型在 MTEB-Code 基准上创下同类模型新纪录，其中 C2LLM-7B 在整体排行榜中位列第一。
原文摘要 · Abstract (English)
We present C2LLM - Contrastive Code Large Language Models, a family of code embedding models in both 0.5B and 7B sizes. Building upon Qwen-2.5-Coder backbones, C2LLM adopts a Pooling by Multihead Attention (PMA) module for generating sequence embedding from token embeddings, effectively 1) utilizing the LLM's causal representations acquired during pretraining, while also 2) being able to aggregate information from all tokens in the sequence, breaking the information bottleneck in EOS-based sequence embeddings, and 3) supporting flexible adaptation of embedding dimension, serving as an alternative to MRL. Trained on three million publicly available data, C2LLM models set new records on MTEB-Code among models of similar sizes, with C2LLM-7B ranking 1st on the overall leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。