arXiv:2510.23893cs.SEcs.AI2025-10被引 3

用大模型实现系统自动互操作,测试显示部分模型效果显著。

Evaluating the effectiveness of LLM-based interoperability

  • 用13个大模型在农业场景测试自动互操作能力
  • qwen2.5-coder:32b在多数数据集上通过率超99%
  • 适合关注自动化系统集成的研究者与工程师

随着系统间动态性和异构性加剧,互操作性面临更大挑战。本文研究大语言模型(LLMs)在运行时实现系统自主互操作的有效性。选取13个开源LLM,在农业互操作场景下构建4个版本数据集,每模型进行3次实验,采用两种策略(DIRECT和CODEGEN)。结果显示,qwen2.5-coder:32b在三个数据集版本中表现最佳,使用DIRECT策略平均pass@1 ≥ 0.99,使用CODEGEN策略平均pass@1 ≥ 0.89。在包含单位转换的第四个数据集版本中,所有DIRECT策略均失败,而CODEGEN策略下qwen2.5-coder:32b仍成功,平均pass@1为0.75。结论表明部分大模型具备自主互操作能力,建议在更多领域进一步评估,并加强可靠性策略研究。

原文摘要 · Abstract (English)

Background: Systems of systems are becoming increasingly dynamic and heterogeneous, and this adds pressure on the long-standing challenge of interoperability. Besides its technical aspect, interoperability has also an economic side, as development time efforts are required to build the interoperability artifacts. Objectives: With the recent advances in the field of large language models (LLMs), we aim at analyzing the effectiveness of LLM-based strategies to make systems interoperate autonomously, at runtime, without human intervention. Method: We selected 13 open source LLMs and curated four versions of a dataset in the agricultural interoperability use case. We performed three runs of each model with each version of the dataset, using two different strategies. Then we compared the effectiveness of the models and the consistency of their results across multiple runs. Results: qwen2.5-coder:32b was the most effective model using both strategies DIRECT (average pass@1 >= 0.99) and CODEGEN (average pass@1 >= 0.89) in three out of four dataset versions. In the fourth dataset version, which included an unit conversion, all models using the strategy DIRECT failed, whereas using CODEGEN qwen2.5-coder:32b succeeded with an average pass@1 = 0.75. Conclusion: Some LLMs can make systems interoperate autonomously. Further evaluation in different domains is recommended, and further research on reliability strategies should be conducted.

大模型系统互操作自动化集成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。