提出四层知识注入框架,提升多模态生成的安全性与准确性。
Where Should Knowledge Enter? A Layered Framework for Knowledge Infusion in Multimodal Iterative Generative Model

- 按生成过程的四个阶段设计知识注入层:表面、轨迹、潜在、参数。
- 实测多层叠加使违反知识的输出减少70.97%,逐层互补有效。
- 适合需要高可靠性的医疗、金融等安全敏感场景使用。
多模态生成模型虽流畅,但在需遵循结构化、领域或安全关键知识时仍不可靠。现有方法如提示增强、引导、潜在编辑或微调,通常按技术分类而非作用于生成流程的具体环节。我们提出,知识注入本质上是干预层级问题——生成过程为内部状态的演化轨迹,知识可作用于输入/输出边界、转移函数、中间状态和模型参数四类结构组件,对应表面、轨迹、潜在、参数四层注入。我们在扩散模型中实现该框架,将代表性方法映射至各层,并推导多层组合的设计原则。在含双扩散模型的多模态知识图谱安全对齐实验中,累计实现表面(输入端与输出端)与轨迹-潜在(生成中段)三层注入。实证表明,每新增一层可解决前层无法覆盖的失败类别,相较原始生成,知识违背输出降低70.97%,验证了框架的互补性预测。
原文摘要 · Abstract (English)
Multimodal generative models produce fluent outputs but remain unreliable when generation must respect structured, domain-specific, or safety-critical knowledge. Existing methods incorporate knowledge through mechanisms such as prompt augmentation, guidance, latent editing, or fine-tuning, yet they are typically categorized by technique rather than by the component of the generative process they modify. We argue that knowledge infusion in iterative generative models is fundamentally anintervention-layer problem. Since thegenerative process unfolds as a trajectory of internal states, knowledge can act on four structurally distinct components of this process: the input/output boundary, the transition function, the intermediate state, and the model parameters. This maps to four intervention layers: surface, trajectory, latent, and parametric infusion. We instantiate the framework in diffusion models, map representative methods to all four layers, and derive design principles for multi-layer composition. In a controlled safety-alignment experiment using a multimodal knowledge graph with two diffusion backbones, we implement three of the four layers cumulatively, surface (input-side and output-side) and trajectory--latent (mid-generation). We show empirically that each additional layer addresses failure classes that prior layers cannot reach, reducing knowledge-violating outputs by 70.97% compared to vanilla generation and empirically confirming the framework's complementarity prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。