arXiv:2604.18203cs.CL2026-04ACL

多模态大模型算数能力差,研究发现计算负荷是关键瓶颈。

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs

论文配图:Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
图 1 · 摘自论文原文
  • 设计跨模态乘法基准,系统控制数字长度与表现形式。
  • 计算负荷(C)越大,准确率越低,C>100时接近零。
  • 模型失败源于计算而非感知,偏好分解式推理策略。

多模态大模型虽能准确识别不同模态中的数值内容,但在面对相同算术问题时,却无法精确完成多位数乘法,无论问题以数字、数词、图像或音频形式呈现。现有评测常缺乏跨模态配对样本,难以真实比较模型间的算术能力边界。为此,我们构建了一个受控的多模态乘法基准,因子化变化数字长度、稀疏性、表示方式(如数字与数词)及模态(文本、渲染图像、音频),并使用可复现生成器提供配对实例。我们定义算术负荷 C 为总位数与非零位数的乘积,作为操作次数的紧凑且机制驱动的代理指标。实验显示,随着 C 增大,准确率急剧下降,通常在 C > 100 时趋近于零。C 对性能具有强预测力,跨模态和模型间 R² 常超过 0.5,接近基于中间步骤计数的复杂度指标。进一步的感知-计算解耦分析表明,多模态性能下降主要源于计算而非感知:在感知匹配测试中,模型跨模态准确率均高于 99%。此外,我们引入强制补全损失探针,评估特定启发式推理前缀(如列式乘法、分配分解、四舍五入补偿)的表现。结果显示,文本与视觉模态均偏好分解策略;针对性的启发式 LoRA 适配器产生近正交更新,但降低准确率,表明基础模型具备良好的内部路由机制。

原文摘要 · Abstract (English)

Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, audio), with paired instances from a reproducible generator. We also define arithmetic load, C, as the product of the total and non-zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C grows, often nearing zero by C > 100. Indeed, C remains predictive of performance across modalities and models, with R-squared often > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched-perception checks, models are near-perfect (> 99%) across modalities, even when multiplication accuracy drops. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a forced-completion loss probe that scores heuristic-specific reasoning prefixes--including columnar multiplication, distributive decomposition, and rounding/compensation. Here, decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router.

多模态算术推理大模型计算负荷

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。