GPT-SW3: An Autoregressive Language Model for the Scandinavian LanguagesVise andre og tillknytning
2024 (engelsk)Inngår i: 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings, European Language Resources Association (ELRA) , 2024, s. 7886-7900Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]
This paper details the process of developing the first native large generative language model for the North Germanic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation, applications, and considerations for release strategies. We discuss pros and cons of developing large language models for smaller languages and in relatively peripheral regions of the globe, and we hope that this paper can serve as a guide and reference for other researchers that undertake the development of large generative models for smaller languages.
sted, utgiver, år, opplag, sider
European Language Resources Association (ELRA) , 2024. s. 7886-7900
Emneord [en]
Computational linguistics; Auto-regressive; Data collection; Development process; Generative model; Language model; Large language model; Low resource languages; Multilinguality; Peripheral regions; Release strategies; Data handling
HSV kategori
Identifikatorer
URN: urn:nbn:se:ri:diva-73777Scopus ID: 2-s2.0-85195971043OAI: oai:DiVA.org:ri-73777DiVA, id: diva2:1877157
Konferanse
Joint 30th International Conference on Computational Linguistics and 14th International Conference on Language Resources and Evaluation, LREC-COLING 2024 Hybrid, Torino, Italy. 20 May 2024 through 25 May 2024
Merknad
The GPT-SW3 initiative has been enabled by the collaboration and support from the following organizations: RISE (collaboration on experiments, data storage and compute), NVIDIA (support with the deduplication code base and Nemo Megatron), Vinnova (funding via contracts 2019-02996, 2020-04658 and 2022-00949), WASP WARA media and language (access to Berzelius via SNIC/NAISS). The computations and data handling were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) and the Swedish National Infrastructure for Computing (SNIC) at Berzelius partially funded by the Swedish Research Council through grant agreements 2022-06725 and 2018-05973. Johan Raber at the National Supercomputer Center is acknowledged for assistance concerning technical and implementational aspects in making the code run on the Berzelius resources
2024-06-252024-06-252025-09-23bibliografisk kontrollert