通过解码轨迹分析,用结构特征区分大模型身份。
Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs
- 提出定向解码图(DDM)捕捉解码步骤间结构关系。
- 用高斯轨迹匹配法提升模型归属识别准确率。
- 适合模型溯源、防伪或版本比对场景使用。
离散扩散语言模型(dLLMs)作为非自回归建模的新范式,具备更快的推理速度和在代码生成与数学任务中的优异表现。本文发现其解码机制不仅能提升模型性能,还可作为强大的模型归属工具。面对不同模型、同一模型不同检查点或备份之间的归属难题,我们提出两个核心问题:如何从解码轨迹中提取有效信息,以及如何高效利用这些信息。我们观察到直接使用每步模型置信度效果不佳,原因在于dLLMs的双向解码特性导致置信度高度冗余,掩盖了解码顺序与依赖关系的结构性信号。为此,我们提出新型信息提取方法——定向解码图(DDM),能更好揭示模型特异性行为。进一步地,我们设计高斯轨迹归属(GTA)方法:为每个目标模型在每个解码位置拟合细胞级高斯分布,将轨迹的对数似然定义为归属得分——若某轨迹在特定模型分布下的似然更高,则更可能由该模型生成。多种设置下的实验验证了方法的有效性。
原文摘要 · Abstract (English)
Discrete Diffusion Large Language Models (dLLMs) have recently emerged as a competitive paradigm for non-autoregressive language modeling. Their distinctive decoding mechanism enables faster inference speed and strong performance in code generation and mathematical tasks. In this work, we show that the decoding mechanism of dLLMs not only enhances model utility but also can be used as a powerful tool for model attribution. A key challenge in this problem lies in the diversity of attribution scenarios, including distinguishing between different models as well as between different checkpoints or backups of the same model. To ensure broad applicability, we identify two fundamental problems: what information to extract from the decoding trajectory, and how to utilize it effectively. We first observe that relying directly on per-step model confidence yields poor performance. This is mainly due to the bidirectional decoding nature of dLLMs: each newly decoded token influences the confidence of other decoded tokens, making model confidence highly redundant and washing out structural signal regarding decoding order or dependencies. To overcome this, we propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors. Furthermore, to make full use of the extracted structural information during attribution, we propose Gaussian-Trajectory Attribution (GTA), where we fit a cell-wise Gaussian distribution at each decoding position for each target model, and define the likelihood of a trajectory as the attribution score: if a trajectory exhibits higher log-likelihood under the distribution of a specific model, it is more likely to have been generated by that model. Extensive experiments under different settings validate the utility of our methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。