用符号化方法让模型理解图表,实现可解释的视觉推理。
Hierarchical Process Reward Models are Symbolic Vision Learners
- 通过分层奖励机制学习点线面关系,构建可执行的符号化编码器。
- 图表重建误差降低98.2%,在多个基准上优于GPT-4o和现有模型。
- 适合需要可解释性与逻辑推理的视觉任务,如数学图表理解。
符号化计算机视觉通过显式的逻辑规则和结构化表示来解析图表,实现机器视觉中的可解释理解,这要求与基于像素的视觉模型截然不同的学习范式。符号化视觉学习者将图表解析为几何原语——点、线、形状,而像素模型则处理纹理和颜色。我们提出一种新颖的自监督符号自动编码器,将图表编码为潜在空间中的结构化原语及其相互关系,并通过可执行引擎解码以重构输入图表。该架构的核心是符号化分层过程奖励建模,通过分步解析奖励强制保证点在线上、线在形上、形在关系上的一致性。由于传统强化学习在图示重构中探索能力差,我们引入稳定机制以平衡探索与利用。我们在下游任务上微调符号编码器,构建一个神经符号系统,通过基于推理的视觉奖励融合神经网络的推理能力与符号模型的可解释性。在重建、感知和推理任务上的评估表明本方法有效:几何图表重建的均方误差降低98.2%,使用7B模型在图表重建上超越GPT-4o 0.6%,在MathGlance感知基准上提升13%,在MathVerse和GeoQA推理基准上分别提升3%。
原文摘要 · Abstract (English)
Symbolic computer vision represents diagrams through explicit logical rules and structured representations, enabling interpretable understanding in machine vision. This requires fundamentally different learning paradigms from pixel-based visual models. Symbolic visual learners parse diagrams into geometric primitives-points, lines, and shapes-whereas pixel-based learners operate on textures and colors. We propose a novel self-supervised symbolic auto-encoder that encodes diagrams into structured primitives and their interrelationships within the latent space, and decodes them through our executable engine to reconstruct the input diagrams. Central to this architecture is Symbolic Hierarchical Process Reward Modeling, which applies hierarchical step-level parsing rewards to enforce point-on-line, line-on-shape, and shape-on-relation consistency. Since vanilla reinforcement learning exhibits poor exploration in the policy space during diagram reconstruction; we thus introduce stabilization mechanisms to balance exploration and exploitation. We fine-tune our symbolic encoder on downstream tasks, developing a neuro-symbolic system that integrates the reasoning capabilities of neural networks with the interpretability of symbolic models through reasoning-grounded visual rewards. Evaluations across reconstruction, perception, and reasoning tasks demonstrate the effectiveness of our approach: achieving a 98.2% reduction in MSE for geometric diagram reconstruction, surpassing GPT-4o by 0.6% with a 7B model on chart reconstruction, and improving by +13% on the MathGlance perception benchmark, and by +3% on MathVerse and GeoQA reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。