arXiv:2608.28980cs.CLcs.AI2026-09

语言模型虽强,但结构化任务仍需专门设计的架构支持。

The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era

  • 用语言模型处理结构数据时,常需额外引入图模块或结构标记来补足缺失结构。
  • 在少样本、符号任务等场景下,语言模型表现接近专用架构,但结构计算能力不足。
  • 结构化任务本质依赖结构表示,语言模型无法完全替代专用架构,适合研究者关注系统设计本质。

本文通过分析2016至2026年间涵盖九种模态的159篇论文,探讨了语言模型能否取代传统结构化数据专用架构。研究区分任务执行与结构表征及计算的保留,将现有方法归纳为八类表征范式,从纯语言模型到全专用架构。发现语言模型在极端少样本预测、离散符号任务、文本标注知识图谱及大规模单模态预训练中表现优异。然而,当直接评估结构表征或计算能力而非仅准确率时,无证据表明语言模型可普遍替代专用架构。跨独立研究社区中普遍存在:当语言不足以完成任务时,会重新引入图模块、结构化标记、专用注意力等非语言组件。因此,专业化更多是转移而非消失。尽管语言模型随规模扩大性能提升,其与结构感知架构之间的差距是否最终可弥合仍未被验证。

原文摘要 · Abstract (English)

Can the specialized architectures that machine learning has traditionally built for structured data be replaced by language-based models? This question is examined through a review of 159 papers (2016--2026) across nine modalities, with predictive accuracy considered alongside structural representation and computation. A distinction is made between performing a task and preserving and computing the structure that makes the task tractable, and existing approaches are organized into eight representational regimes, ranging from language-only systems to fully specialized architectures. Language-mediated models are found to be highly competitive in specific settings, including extreme few-shot prediction, discretized symbolic tasks, textually annotated knowledge graphs, and large-scale single-modality pretraining. However, whenever structural representation or computation is directly evaluated rather than accuracy alone, no evidence of general architectural replacement is found. Instead, a recurring pattern is observed across independent research communities: when language alone is insufficient, the missing structure is reintroduced through a graph module, structural tokens, specialized attention, or another non-linguistic component. In this sense, specialization more often relocates than disappears. Moreover, although performance of language-based models is improved by scaling, whether the gap to a structure-aware architecture can eventually be eliminated remains untested.

基础模型结构化数据架构设计语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。