通过几何特征选择最佳位置，用扩散模型替换语言模型底层状态
Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
- 基于隐藏状态几何特征筛选适合插入扩散模型的层
- 在80亿参数模型上，状态重建效果优于传统连续扩散基线
- 适合研究模型融合、扩散与语言模型结合的学者
连续扩散语言模型性能落后于自回归变换器，部分原因在于扩散过程作用于不适宜语言去噪和词元恢复的空间。本文提出DiHAL，一种基于几何引导的扩散-变换器混合模型，探索扩散应插入预训练变换器的何处。DiHAL使用几何代理评分各层，选定适合扩散的隐藏状态接口，并将下层变换器前缀替换为扩散桥，保留上层结构与原始语言模型头。通过重构选定层的隐藏状态而非直接进行连续到离散的词元恢复，DiHAL避免了直接映射难题。在80亿参数骨干模型上的实验表明，几何分数能有效预测在固定桥接训练协议下的浅层插入位置；且在匹配扩散/恢复训练预算的诊断对比中，隐藏状态恢复性能优于连续扩散基线。结果表明，隐藏状态几何特性有助于识别预训练语言模型内部扩散替换的可行位置。
原文摘要 · Abstract (English)
Continuous diffusion language models lag behind autoregressive transformers, partly because diffusion is applied in spaces poorly suited to language denoising and token recovery. We propose DiHAL, a geometry-guided diffusion-transformer hybrid that asks where diffusion should enter a pretrained transformer. DiHAL scores layers with geometry-based proxies, selects a diffusion-friendly hidden-state interface, and replaces the lower transformer prefix with a diffusion bridge while retaining the upper layers and original LM head. By reconstructing the selected-layer hidden state rather than tokens, DiHAL avoids direct continuous-to-discrete recovery. Experiments on 8B-scale backbones show that the geometry score predicts effective shallow insertion layers under a fixed bridge-training protocol and that hidden-state recovery improves over continuous diffusion baselines in a diagnostic comparison matching the diffusion/recovery training budget. These results suggest that hidden-state geometry helps identify where diffusion-based replacement is feasible inside pretrained language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。