为泰铢金融领域定制大模型，解决术语和法规本地化难题。
Thai Financial Domain Adaptation of THaLLE -- Technical Report
- 用泰国证券交易所考题数据+数据增强训练专用模型
- 在三级金融考试中最高达84%准确率，表现优于通用模型
- 适合需要泰语金融咨询能力的机构或开发者使用
大型语言模型在通用任务上表现优异，但在专业领域如金融中面临术语和本地法规挑战。现有金融大模型如FinGPT和BloombergGPT缺乏对泰铢金融领域的支持。本文基于泰国证券交易所投资顾问（IC）考试数据集，构建了泰语金融大模型。针对数据量不足问题，采用数据增强、ReLoRA高效训练、持续预训练（CPT）注入领域知识，并运用秩稳定低秩适配（rsLoRA）进行微调。通过监督微调（SFT）模拟真实考试场景，利用直接偏好优化（DPO）基于反馈优化模型。模型在IC考试的P1、P2、P3级别分别取得72%、72%、84%的准确率，验证了其在泰语金融咨询任务中的有效性，具备面向垂直应用的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。