用检索增强生成自动写对象约束语言规则,选对检索方法很关键。
Optimizing Retrieval Augmented Generation for Object Constraint Language
- 对比三种检索方式:词汇、语义和稀疏向量,优化上下文获取
- SPLADE在低检索数下表现最佳,过度检索反而降低准确率
- 适合想自动化生成复杂系统约束规则的研究者
对象约束语言(OCL)在基于模型的系统工程中用于定义精确约束,但手动编写耗时费力。本研究探索检索增强生成(RAG)在自动化生成OCL规则中的优化,重点分析不同检索策略的影响。评估了三种方法:基于词汇的BM25、基于BERT的语义检索和基于SPLADE的稀疏向量检索,并分析其为大语言模型提供相关上下文的有效性。进一步与当前最优的图结构方法PathOCL进行直接对比,在统一评估框架下分析各检索方法的表现。实验结果表明,虽然检索能提升生成准确性,但效果依赖于检索方法和检索块数(k)。BM25表现低于基线,而语义方法(BERT和SPLADE)更优,其中SPLADE在较低k值下表现最佳;但高k值会导致无关内容被引入,损害模型性能。研究强调需优化检索配置以平衡上下文相关性和输出一致性,为利用RAG改进OCL生成提供了重要参考。
原文摘要 · Abstract (English)
The Object Constraint Language (OCL) is essential for defining precise constraints within Model-Based Systems Engineering (MBSE). However, manually writing OCL rules is complex and time-consuming. This study explores the optimization of Retrieval-Augmented Generation (RAG) for automating OCL rule generation, focusing on the impact of different retrieval strategies. We evaluate three retrieval approaches $\unicode{x2013}$ BM25 (lexical-based), BERT-based (semantic retrieval), and SPLADE (sparse-vector retrieval) $\unicode{x2013}$ analyzing their effectiveness in providing relevant context for a large language model. To further assess our approach, we compare and benchmark our retrieval-optimized generation results against PathOCL, a state-of-the-art graph-based method. We directly compare BM25, BERT, and SPLADE retrieval methods with PathOCL to understand how different retrieval methods perform for a unified evaluation framework. Our experimental results, focusing on retrieval-augmented generation, indicate that while retrieval can enhance generation accuracy, its effectiveness depends on the retrieval method and the number of retrieved chunks (k). BM25 underperforms the baseline, whereas semantic approaches (BERT and SPLADE) achieve better results, with SPLADE performing best at lower k values. However, excessive retrieval with high k parameter can lead to retrieving irrelevant chunks which degrades model performance. Our findings highlight the importance of optimizing retrieval configurations to balance context relevance and output consistency. This research provides insights into improving OCL rule generation using RAG and underscores the need for tailoring retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。