用AI代理自动生成税务软件测试用例，提升法律关键系统可靠性
An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software
- 设计多智能体框架，用LLM自动生成符合税法逻辑的测试用例
- 在复杂税务任务中，小模型GPT-4o-mini达45%最差通过率，优于大模型
- 适合法律自动化、可信AI开发人员参考，解决法律文本执行难题
大型语言模型（LLMs）在将自然语言法规转换为可执行逻辑方面展现潜力，但在法律关键场景中仍受模糊性和幻觉影响。本文以美国联邦税务申报为例，提出一种智能体方法来开发法律关键软件。核心挑战在于测试用例生成中的“预言机问题”——正确输出需依赖对法律的解读。基于变异测试思想，引入高阶变异关系，比较相似个体在结构化变动下的系统输出。由于手工编写此类关系繁琐易错，我们采用由LLM驱动的角色化框架，实现测试生成与代码合成自动化。构建了多智能体系统，将税法转化为可执行软件，并集成变异测试智能体以搜寻反例。实验表明，使用较小模型GPT-4o-mini的框架在复杂税法任务中达到45%的最低通过率，优于前沿模型GPT-4o和Claude 3.5的9%-15%。结果支持基于智能体的LLM方法是实现从自然语言规范出发的稳健、可信法律关键软件的有效路径。
原文摘要 · Abstract (English)
Large language models (LLMs) show promise for translating natural-language statutes into executable logic, but reliability in legally critical settings remains challenging due to ambiguity and hallucinations. We present an agentic approach for developing legal-critical software, using U.S. federal tax preparation as a case study. The key challenge is test-case generation under the oracle problem, where correct outputs require interpreting law. Building on metamorphic testing, we introduce higher-order metamorphic relations that compare system outputs across structured shifts among similar individuals. Because authoring such relations is tedious and error-prone, we use an LLM-driven, role-based framework to automate test generation and code synthesis. We implement a multi-agent system that translates tax code into executable software and incorporates a metamorphic-testing agent that searches for counterexamples. In experiments, our framework using a smaller model (GPT-4o-mini) achieves a worst-case pass rate of 45%, outperforming frontier models (GPT-4o and Claude 3.5, 9-15%) on complex tax-code tasks. These results support agentic LLM methodologies as a path to robust, trustworthy legal-critical software from natural-language specifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。