通过可重构图像验证模型共识与正确性的关系,发现共识并非总代表正确。
When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering
- 用程序生成语义等价图像集,精准定位模型错误。
- 共识在高精度时有效,但过度追求一致性反而降低准确率。
- 绘图库差异是误差主因,远超其他因素影响。
模型在扰动输入下的一致性常被用作可靠性信号和自训练目标,前提是这种一致性能反映正确性。然而,这种关联极少被直接测量:自然图像扰动仅假设保留语义,且无精确答案可定位错误。科学图表克服了这两个障碍——图表由程序生成,重绘后语义保持一致,并拥有程序化精确的答案。我们构建了 RENDEQ,用于生成此类渲染等价图像集,并在三个开源视觉语言模型上测量该耦合关系,每项发现均经三次独立实例验证。重渲染优于重采样,在准确性和可靠性上均表现更优。共识优于携带证据的基线(平均词元对数概率)在三个模型中的两个,第三例持平,逆转了此前因渲染管道故障导致的错误复现结果。误差分布主要集中于一个风格因子——绘图库,其影响超过次大因子两倍以上,且比噪声水平高一个数量级。在模型自身跨渲染共识上进行微调后,五个复现实验中准确率全部下降,与自然图像上的已有结果相反。共识仅在模型错误分布足够分散时才表明正确性;而奖励共识的目标恰恰破坏了这种分散性。
原文摘要 · Abstract (English)
A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。