arXiv:2605.08526cs.LG2026-05

让智能体学会稳定执行动作,通过分离文字和视觉技能。

Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck

  • 用条件多模态信息瓶颈分离可解释的文本技能与残留感知信息。
  • 在多个任务上提升动作一致性,且无需多次采样推理开销。
  • 适合需要高可靠性的多模态智能体系统,如机器人控制。

基于大语言模型的智能体虽能规划并执行长序列动作,但其执行过程在不同试验间常不一致,影响可靠性。提升一致性需将试错轨迹提炼为可复用的技能,保留任务相关不变量,同时去除轨迹特异性噪声。然而,在多模态场景下,关键挑战在于有用不变量分布在视觉与语言信息中,且不同模态支持不同类型的可复用技能内容:部分技能可被语言描述,另一些则存在于文本无法表达的感知证据中。仅使用文本技能会丢失感知线索,而盲目存储文本与感知信息则引入冗余和噪声。现有推理时方法(如自一致性)通过昂贵的多样本解码提升可靠性,而内部化策略缺乏区分可言说技能与残余感知信息的能力。为此,我们提出条件多模态信息瓶颈(CMIB),用于多模态技能构建。CMIB从联合瓶颈出发,推导出精确的顺序分解:(1) 文本阶段瓶颈提炼可解释的技能卡片;(2) 条件多模态瓶颈仅压缩在文本之外仍具预测性的残余感知信息。不同于简单的双流结构,CMIB显式将多模态潜在表示条件于文本技能,从而结构性地减少跨模态冗余,并实现对文本与感知压缩的独立控制。我们以变分目标实例化CMIB,使其条件分解在优化上可行,生成可复用的多模态技能,在不增加多样本推理开销的前提下提升执行稳定性。

原文摘要 · Abstract (English)

While LLM-based agents excel at planning and executing long action sequences, their execution often remains inconsistent across trials, limiting reliability. Consolidating agent consistency requires distilling trial-error trajectories into reusable skills that preserve task-relevant invariants while discarding trajectory-specific noise. However, in multimodal settings, the key challenge is not only that useful invariants are distributed across vision and language information, but that different modalities support different kinds of reusable skill content: while some skills are verbalizable and interpretable, others reside in perceptual evidence beyond text. Text-only skills may lose perceptual cues, whereas storing text and perception naively introduces redundancy and noise. Existing inference-time methods, such as self-consistency, improve reliability through costly multi-sample decoding, while internalization strategies lack a way to separate verbalizable skill content from residual perceptual information. To address this, we introduce Conditional Multimodal Information Bottleneck (CMIB), a method for multimodal skill construction. CMIB begins with a joint bottleneck over multimodal skills and derives an exact sequential decomposition: (1) a text-stage bottleneck distilling interpretable skill cards, and (2) a conditional multimodal bottleneck compressing only residual information in perception that remains predictive beyond text. Unlike naive two-stream formulations, CMIB explicitly conditions the multimodal latent on the text skill, thus structurally reducing cross-modal redundancy and enabling independent control over textual and perceptual compression. We instantiate CMIB with a variational objective that makes its conditional decomposition tractable to optimize, yielding reusable multimodal skills that improve execution stability without incurring multi-sample inference overhead.

多模态智能体技能学习信息瓶颈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。