测试大模型对消息序列图语义的理解能力，发现仅勉强掌握基础概念。
(How) Do Large Language Models Understand High-Level Message Sequence Charts?
- 用129个任务测试三款大模型对高阶消息序列图的语义理解能力
- 整体准确率仅52%，抽象与组合类任务准确率低至36%
- 对因果依赖和共区域等高级概念几乎无法理解，适合软件形式化研究者参考
大型语言模型（LLMs）正被广泛用于软件开发全周期的任务自动化，但其处理设计文档时是否保持语义一致性仍不明确，尤其在架构设计规范方面研究不足。本文聚焦高阶消息序列图（HMSCs），这类图形化模型具有严格的数学语义，是统一建模语言（UML）中序列图的基础。我们通过129项语义任务评估三款模型（Gemini-3、GPT-5.4、Qwen-3.6）对HMSCs的理解能力，涵盖从基本元素查询（如事件及其顺序）到语义保持的抽象与组合、迹集计算及迹等价的标记转移系统（LTS）构造。结果显示，模型整体准确率为52%，其中基础语义理解达88%，但在抽象与组合任务中降至36%，迹与LTS相关任务仅为42%。所有模型均未能正确运用共区域和显式因果依赖概念进行语义保持转换。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are being employed widely to automate tasks across the software development life-cycle. It is, however, unclear whether these tasks are performed consistently with respect to the semantics of the artefacts being handled. This question is particularly under-researched concerning architectural design specification. In this paper, we address this question for High-Level Message Sequence Charts (HMSCs). These are visual models with a rigorous formal semantics that have been used for various purposes, including as a foundation for Sequence Diagrams in the Unified Modelling Language (UML). We examine whether LLMs "understand" the semantics of HMSCs by examining three LLMs (Gemini-3, GPT-5.4, and Qwen-3.6) on how they perform 129 semantic tasks ranging from querying basic semantic constructs in HMSCs (i.e., events and their ordering) to semantic-preserving abstractions and compositions, and calculating the set of traces and trace-equivalent labelled transition systems. The results show that LLMs only have a modest understanding of the formal semantics of HMSCs (ca. 52% overall accuracy), with great variability across different semantic concepts: while LLMs seem to understand the basic semantic concepts of MSCs (ca. 88% accuracy), they struggle with semantic reasoning in tasks involving abstraction and composition (ca. 36% accuracy) and traces and LTSs (ca. 42% accuracy). In particular, all three LLMs struggle with the notions of co-region and explicit causal dependencies and never employed them in semantic-preserving transformations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。