测试GPT模型在日籍注册营养师考试中的助考表现，部分模型勉强达标但稳定性差。
Evaluation of GPT-based large language generative AI models as study aids for the national licensure examination for registered dietitians in Japan
- 用日本营养师执照题测试ChatGPT和Bing系列模型，评估其答题能力。
- Bing-Precise（66.2%）和Bing-Creative（61.4%）超过及格线，其余未达标。
- 模型答案不一致，提示工程改善有限，不适合可靠备考使用。
基于大语言模型（LLM）的生成式人工智能（如ChatGPT）在医学、教育等领域取得显著进展，但在营养学教育，尤其是日本注册营养师国家考试中的表现仍缺乏研究。本研究旨在评估当前LLM生成式AI作为营养专业学生备考工具的潜力。使用日本注册营养师国家考试真题作为提示输入ChatGPT及三个Bing模型（Precise、Creative、Balanced），基于GPT-3.5与GPT-4。每道题在独立会话中测试，分析准确率、一致性与响应时间。额外尝试角色设定等提示工程以提升性能。结果显示，Bing-Precise（66.2%）和Bing-Creative（61.4%）超过60%及格线，而Bing-Balanced（43.3%）和ChatGPT（42.8%）未达标。两者在多数科目表现优于其他模型，但在营养教育领域所有模型均表现不佳。所有模型在重复测试中未能稳定输出相同正确答案，表明答案一致性差。ChatGPT虽模式较一致，但准确率低。提示工程仅在明确提供正确答案与解释时带来小幅提升。尽管个别模型略超及格线，整体准确率与稳定性仍不理想。所有模型在一致性与鲁棒性方面存在明显局限，需进一步改进才能成为可靠的执照备考辅助工具。
原文摘要 · Abstract (English)
Generative artificial intelligence (AI) based on large language models (LLMs), such as ChatGPT, has demonstrated remarkable progress across various professional fields, including medicine and education. However, their performance in nutritional education, especially in Japanese national licensure examination for registered dietitians, remains underexplored. This study aimed to evaluate the potential of current LLM-based generative AI models as study aids for nutrition students. Questions from the Japanese national examination for registered dietitians were used as prompts for ChatGPT and three Bing models (Precise, Creative, Balanced), based on GPT-3.5 and GPT-4. Each question was entered into independent sessions, and model responses were analyzed for accuracy, consistency, and response time. Additional prompt engineering, including role assignment, was tested to assess potential performance improvements. Bing-Precise (66.2%) and Bing-Creative (61.4%) surpassed the passing threshold (60%), while Bing-Balanced (43.3%) and ChatGPT (42.8%) did not. Bing-Precise and Bing-Creative generally outperformed others across subject fields except Nutrition Education, where all models underperformed. None of the models consistently provided the same correct responses across repeated attempts, highlighting limitations in answer stability. ChatGPT showed greater consistency in response patterns but lower accuracy. Prompt engineering had minimal effect, except for modest improvement when correct answers and explanations were explicitly provided. While some generative AI models marginally exceeded the passing threshold, overall accuracy and answer consistency remained suboptimal. Moreover, all the models demonstrated notable limitations in answer consistency and robustness. Further advancements are needed to ensure reliable and stable AI-based study aids for dietitian licensure preparation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。