通过语义对齐实现跨模型细粒度知识迁移,突破参数不兼容瓶颈。
Beyond Neural Incompatibility: Cross-Scale Knowledge Transfer in Language Models through Latent Semantic Alignment
- 用激活值作为媒介,通过语义分解与重构实现跨尺度知识迁移。
- 在4个基准上验证有效,浅层到深层迁移仅训练前沿层,性能提升显著。
- 适合做模型轻量化或低资源场景下的知识复用,尤其关注高效迁移的团队。
语言模型(LMs)在其参数中编码了大量知识,但如何实现细粒度的参数知识迁移(PKT)仍不明确。当源模型与目标模型在架构和参数化上存在差异时,直接参数复用受限于神经不兼容性。本文提出,潜在语义对齐是跨尺度知识迁移的关键前提。不同于直接迁移层参数,该方法以激活值为传输介质。 extsc{SemAlign}包含两个阶段:层归因阶段识别任务相关的源层,并为每个目标层选择唯一对应源层;语义对齐阶段逐层配对并利用源侧语义监督优化目标模型。对齐在潜在空间中通过语义分解与重构完成。浅层到深层迁移中仅训练前沿目标层,层目标通过匹配中心化词-词关系几何结构来监督残差贡献,输出KL损失则保持源级预测行为。因此,传输媒介并非参数块或绝对隐藏状态,而是由配对源层监督诱导的目标空间残差几何。在四个基准上的评估证明了 extsc{SemAlign}的有效性,进一步分析确认语义分解与重构提供了稳定的跨尺度知识迁移机制。
原文摘要 · Abstract (English)
Language Models (LMs) encode substantial knowledge in their parameters, yet it remains unclear how to transfer such knowledge in a fine-grained manner, namely parametric knowledge transfer (PKT). A central challenge is to make cross-scale transfer effective and efficient when source and target models differ in architecture and parameterization, making direct parameter reuse strongly limited by neural incompatibility. In this paper, we identify latent semantic alignment as the key prerequisite for cross-scale knowledge transfer. Instead of directly moving layer parameters, our approach uses activations as the transfer medium. \textsc{SemAlign} has two stages: an \emph{layer attribution} stage that attributes task-relevant source layers and selects exactly one source layer for each target layer, and a \emph{semantic alignment} stage that pairs them layer by layer and optimizes the target with source-side semantic supervision. The alignment is carried out in latent space through semantic decomposition and recomposition. During the shallow-to-deep transfer, only the frontier target layer is trainable. The layer objective supervises the residual contribution of that layer by matching centered token-token relation geometry against an aligned supervisory residual, while output KL preserves source-level predictive behavior. The transferred medium is therefore neither a parameter block nor an absolute hidden state, but target-space residual geometry induced by paired source-layer supervision. Evaluations on four benchmarks demonstrate the efficacy of \textsc{SemAlign}, and further analysis confirms that semantic decomposition and recomposition provide a stable mechanism for cross-scale knowledge transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。