用人工语言探秘大模型对语言规则的理解能力
Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs
- 构建可交互的智能系统，分模块生成音系、形态、句法等语言要素
- 不同大模型在常见语法模式上表现更好，罕见模式则明显逊色
- 适合研究语言认知、提示工程或对语言学感兴趣的开发者
我们提出一个名为IASC（用于人工语言的交互式智能系统）的框架，利用大语言模型构建人工语言（ConLang）。该系统模块化设计，通过针对性提示分别生成音系、形态、句法、词汇、书写系统和语法手册。其代理式机制允许各模块根据前序步骤的自动生成反馈进行优化。研究目标有二：一是为创造人工语言提供有趣高效的工具；二是利用该框架探索大模型对语言本质与语言概念的认知深度——而非特定语言知识。实验聚焦形态句法模块，发现不同大模型间能力差异显著，且对类型学上常见的语言结构处理更优，稀有结构则表现较差。所有代码已开源。
原文摘要 · Abstract (English)
We present a system that uses LLMs as a tool in the development of Constructed Languages -- ConLangs, which we call IASC (Interactive Agentic System for ConLangs). The system is modular in that it creates each of the components -- phonology, morphology and syntax, lexicon, orthography, and grammatical handbook, using module-specific sets of prompts. The approach is agentic in that various modules allow for refining the output given automatically-generated commentary on a previous step. Our main goals are twofold. First, we aim to provide tools that facilitate an engaging and enjoyable experience in creating artificially constructed languages. Second, the focus of this paper is on using our ConLang framework as a novel way to explore what LLMs 'know' about language -- not what they know about any particular language or encyclopedic facts, but how much they know about and understand language and linguistic concepts. In the experiments, we particularly focus on the morphosyntax module and show that there is a fairly wide gulf in capabilities both among different LLMs and among different linguistic specifications, with it being notably easier for systems to deal with more typologically common patterns than rarer ones. All code is released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。