研究大模型如何在文本与数字冲突时做判断，发现其依赖直觉而非逻辑。
When Text and Numbers Disagree: Evidence Arbitration in Large Language Models
- 设计合成数据集，精确控制文本、数字、时间、来源等变量
- 模型更信近期信息，常盲从外部预测而忽略上下文证据
- 揭示当前大模型融合多源信息时存在系统性偏差
大型语言模型（LLMs）越来越多地应用于文本摘要、数值观测和外部工具输出可能冲突的场景。本文研究当这些证据支持相反决策时，模型如何进行仲裁。为此，我们引入一个受控的合成基准，其中潜在风险轨迹同时生成数值时间序列和自然语言摘要，可构造出仅一种证据源与真实标签一致的情况。该设计允许独立操控模态、时间新近性、来源可靠性及证据出处。在多个开源指令微调模型上测试发现，仲裁行为具有系统性而非随机性：模型表现出明显的文本或数字偏好；对时间新近性的遵循强于显式可靠性提示；甚至会过度依赖外部预测，即使其与直接上下文证据相悖。结果表明，当前大模型在整合异构证据时往往依赖启发式策略，暴露出工具增强型决策系统的潜在失效模式。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used in settings where textual summaries, numerical observations, and external tool outputs may provide conflicting evidence. We study how LLMs arbitrate between such sources when they support opposing decisions. To do so, we introduce a controlled synthetic benchmark in which latent risk trajectories generate both numerical time series and natural language summaries, allowing us to construct conflicts where exactly one evidence source is aligned with the ground-truth label. This design lets us independently manipulate modality, temporal recency, source reliability, and evidence provenance. Across open-weight instruction-tuned models, we find that arbitration behaviour is systematic rather than random: models exhibit distinct text-versus-number preferences, follow temporal recency more consistently than explicit reliability cues, and can over-rely on external forecasts even when they conflict with direct contextual evidence. These results suggest that current LLMs often rely on heuristic arbitration strategies when integrating heterogeneous evidence, highlighting a failure mode for tool-augmented decision systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。