揭示扩散模型重复生成根源,提出可插拔缓解方案
Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective
- 从信息流动视角分析上下文词元作为语义锚点的作用
- 发现深层网络中上下文熵收敛失败导致重复生成
- 提出CoTA方法,提升注意力并抑制不确定输出
基于扩散的多模态大模型(dMLLMs)因推理延迟高而依赖缓存加速解码,但缓存常引发重复文本生成,我们称之为“重复诅咒”。本文从信息流角度分析该现象,发现:(1) 上下文词元聚合语义信息作为锚点,引导最终预测;(2) 信息跨层传播时,上下文词元熵在深层趋于收敛,反映模型预测确信度上升;(3) 重复通常由上下文信息流中断及深层熵无法收敛所致。基于此,我们提出可插拔方法CoTA:增强上下文词元注意力以保持内在信息流模式,并在解码中引入置信度惩罚项,避免受不确定上下文驱动的输出。大量实验表明,CoTA显著缓解重复问题,并在通用任务上实现一致性能提升。代码见https://github.com/ErikZ719/CoTA
原文摘要 · Abstract (English)
Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。