arXiv:2510.11190cs.CV2025-10NeurIPS被引 1

让多模态大模型自由调节联想推理强度,兼顾真实与创意。

FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models

  • 通过分析中间层表示,发现可调控联想倾向。
  • 利用幻觉生成引导向量,提升创造力5.8倍,幻觉率降29%。
  • 无需训练,轻量适配不同创意任务,适合内容生成场景。

多模态大语言模型(MLLMs)在忠实性与创造性之间存在固有权衡,不同任务需不同程度的联想推理。现有方法缺乏灵活调节能力,限制了模型在事实性与创造性场景下的适应性。本文研究了MLLM中联想行为的内部机制,发现:(1) 中间层对联想倾向起关键作用;(2) 修改这些层的表示可有效调节联想强度;(3) 可利用幻觉生成引导向量实现调制。基于此,提出无需训练、轻量级的柔性联想控制框架FlexAC。该方法先诱导幻觉引导的中间表示以编码联想方向,再选取高联想样本构建有效引导向量,并自适应校准其强度以平衡创意与输出稳定性。同时,针对联想的多维特性,通过少量目标域样本前向传播生成任务特定的联想向量,使模型能遵循多样联想方向,更好适应创意任务。实验显示,该方法在Creation-MMBench上创造力提升5.8倍,在CHAIR上幻觉率降低29%,显著优于现有基线,验证了其在柔性控制联想推理方面的有效性。代码已开源。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) face an inherent trade-off between faithfulness and creativity, as different tasks require varying degrees of associative reasoning. However, existing methods lack the flexibility to modulate this reasoning strength, limiting MLLMs' adaptability across factual and creative scenarios. To bridge this gap, we propose equipping MLLMs with mechanisms that enable flexible control over associative reasoning. We begin by investigating the internal mechanisms underlying associative behavior in MLLMs and find that: (1) middle layers play a pivotal role in shaping model's associative tendencies, (2) modifying representations in these layers effectively regulates associative reasoning strength, and (3) hallucinations can be exploited to derive steering vectors that guide this modulation. Building on these findings, we introduce Flexible Association Control (FlexAC), a lightweight and training-free framework for modulating associative behavior in MLLMs. FlexAC first induces hallucination-guided intermediate representations to encode associative directions. Then, it selects high-association instances to construct effective associative steering vectors, whose strengths are adaptively calibrated to balance creative guidance with output stability. Finally, recognizing the multi-dimensional nature of associative reasoning, FlexAC incorporates task-specific associative vectors derived from a forward pass on a few target-domain samples, enabling models to follow diverse associative directions and better adapt to creative tasks. Notably, our method achieves up to a 5.8x improvement in creativity on Creation-MMBench and a 29% reduction in hallucination rate on CHAIR, surpassing existing baselines and demonstrating its effectiveness in enabling flexible control over associative reasoning in MLLMs. Our code is available at https://github.com/ylhz/FlexAC.

多模态联想推理可控生成轻量框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。