arXiv:2511.15717cs.AIcs.CV2025-11

研究不同模态如何影响模型对图像网格的感知与推理,发现图文结合可提升准确率。

How Modality Shapes Perception and Reasoning: A Study of Error Propagation in ARC-AGI

  • 用九种文本与图像模态分离感知与推理,量化误差来源。
  • 图文结合使执行准确率提升约8个百分点,中位相似度达0.20。
  • 适合关注多模态融合与错误归因的研究者阅读。

ARC-AGI 和 ARC-AGI-2 在小色阶量化网格上衡量组合泛化能力,其奖项竞赛将这些更难的保留任务进展视为系统泛化的有意义代理。近期指令优先系统将网格转换为简洁的自然语言或领域特定语言规则,通过生成-执行-选择循环执行,但缺乏对编码如何塑造模型感知以及如何区分指令错误与执行错误的原理性解释。我们假设模态引入感知瓶颈——文本将二维结构扁平化为一维标记,而图像虽保留布局但可能产生补丁尺寸混叠——从而影响哪些网格特征能被可靠感知。为此,我们使用加权集合不一致度量和两阶段推理流程,在九种文本与图像模态中分离感知与推理,发现结构化文本在稀疏特征上可精确定位坐标,图像捕捉二维形状但对分辨率敏感,两者结合可提升执行效果(约8个感知得分;约0.20中位相似度)。总体而言,使表示与Transformer归纳偏置对齐,并实现文本与图像间的交叉验证,可在不改变基础模型的前提下获得更准确的指令和更可靠的执行。

原文摘要 · Abstract (English)

ARC-AGI and ARC-AGI-2 measure generalization-through-composition on small color-quantized grids, and their prize competitions make progress on these harder held-out tasks a meaningful proxy for systematic generalization. Recent instruction-first systems translate grids into concise natural-language or DSL rules executed in generate-execute-select loops, yet we lack a principled account of how encodings shape model perception and how to separate instruction errors from execution errors. We hypothesize that modality imposes perceptual bottlenecks -- text flattens 2D structure into 1D tokens while images preserve layout but can introduce patch-size aliasing -- thereby shaping which grid features are reliably perceived. To test this, we isolate perception from reasoning across nine text and image modalities using a weighted set-disagreement metric and a two-stage reasoning pipeline, finding that structured text yields precise coordinates on sparse features, images capture 2D shapes yet are resolution-sensitive, and combining them improves execution (about 8 perception points; about 0.20 median similarity). Overall, aligning representations with transformer inductive biases and enabling cross-validation between text and image yields more accurate instructions and more reliable execution without changing the underlying model.

多模态推理误差感知建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。