Change search
Link to record
Permanent link

Direct link
Publications (10 of 62) Show all publications
Dahlberg, S., Axelsson, S., Gyllensten, A., Sahlgren, M., Ekgren, A., Holmberg, S. & Andersson Schwarz, J. (2023). A Distributional Semantic Online Lexicon for Linguistic Explorations of Societies. Social science computer review, 41(2), 308-329
Open this publication in new window or tab >>A Distributional Semantic Online Lexicon for Linguistic Explorations of Societies
Show others...
2023 (English)In: Social science computer review, ISSN 0894-4393, E-ISSN 1552-8286, Vol. 41, no 2, p. 308-329Article in journal (Refereed) Published
Abstract [en]

Linguistic Explorations of Societies (LES) is an interdisciplinary research project with scholars from the fields of political science, computer science, and computational linguistics. The overarching ambition of LES has been to contribute to the survey-based comparative scholarship by compiling and analyzing online text data within and between languages and countries. To this end, the project has developed an online semantic lexicon, which allows researchers to explore meanings and usages of words in online media across a substantial number of geo-coded languages. The lexicon covers data from approximately 140 language–country combinations and is, to our knowledge, the most extensive free research resource of its kind. Such a resource makes it possible to critically examine survey translations and identify discrepancies in order to modify and improve existing survey methodology, and its unique features further enable Internet researchers to study public debate online from a comparative perspective. In this article, we discuss the social scientific rationale for using online text data as a complement to survey data, and present the natural language processing–based methodology behind the lexicon including its underpinning theory and practical modeling. Finally, we engage in a critical reflection about the challenges of using online text data to gauge public opinion and political behavior across the world. © The Author(s) 2022.

Place, publisher, year, edition, pages
SAGE Publications Inc., 2023
Keywords
comparative surveys, distributional semantics, language use, natural language processing, semantic similarities, word2vec
National Category
Communication Studies
Identifiers
urn:nbn:se:ri:diva-59347 (URN)10.1177/08944393211049774 (DOI)2-s2.0-85130070813 (Scopus ID)
Available from: 2022-06-20 Created: 2022-06-20 Last updated: 2023-07-06Bibliographically approved
Lenci, A., Sahlgren, M., Jeuniaux, P., Cuba Gyllensten, A. & Miliani, M. (2022). A comparative evaluation and analysis of three generations of Distributional Semantic Models. Language resources and evaluation, 56, 1219
Open this publication in new window or tab >>A comparative evaluation and analysis of three generations of Distributional Semantic Models
Show others...
2022 (English)In: Language resources and evaluation, ISSN 1574-020X, E-ISSN 1574-0218, Vol. 56, p. 1219-Article in journal (Refereed) Published
Abstract [en]

Distributional semantics has deeply changed in the last decades. First, predict models stole the thunder from traditional count ones, and more recently both of them were replaced in many NLP applications by contextualized vectors produced by neural language models. Although an extensive body of research has been devoted to Distributional Semantic Model (DSM) evaluation, we still lack a thorough comparison with respect to tested models, semantic tasks, and benchmark datasets. Moreover, previous work has mostly focused on task-driven evaluation, instead of exploring the differences between the way models represent the lexical semantic space. In this paper, we perform a large-scale evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. First of all, we investigate the performance of embeddings in several semantic tasks, carrying out an in-depth statistical analysis to identify the major factors influencing the behavior of DSMs. The results show that (i) the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous and (ii) static DSMs surpass BERT representations in most out-of-context semantic tasks and datasets. Furthermore, we borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models. RSA reveals important differences related to the frequency and part-of-speech of lexical items. © 2022, The Author(s).

Place, publisher, year, edition, pages
Springer Science and Business Media B.V., 2022
Keywords
Contextual embeddings, Distributional semantics, Evaluation, Representational Similarity Analysis
National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:ri:diva-58900 (URN)10.1007/s10579-021-09575-z (DOI)2-s2.0-85125439429 (Scopus ID)
Available from: 2022-03-25 Created: 2022-03-25 Last updated: 2023-07-07Bibliographically approved
Gogoulou, E., Ekgren, A., Isbister, T. & Sahlgren, M. (2022). Cross-lingual Transfer of Monolingual Models. In: 2022 Language Resources and Evaluation Conference, LREC 2022: . Paper presented at 13th International Conference on Language Resources and Evaluation Conference, LREC 2022, 20 June 2022 through 25 June 2022 (pp. 948-955). European Language Resources Association (ELRA)
Open this publication in new window or tab >>Cross-lingual Transfer of Monolingual Models
2022 (English)In: 2022 Language Resources and Evaluation Conference, LREC 2022, European Language Resources Association (ELRA) , 2022, p. 948-955Conference paper, Published paper (Refereed)
Abstract [en]

Recent studies in cross-lingual learning using multilingual models have cast doubt on the previous hypothesis that shared vocabulary and joint pre-training are the keys to cross-lingual generalization. We introduce a method for transferring monolingual models to other languages through continuous pre-training and study the effects of such transfer from four different languages to English. Our experimental results on GLUE show that the transferred models outperform an English model trained from scratch, independently of the source language. After probing the model representations, we find that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in English. ©  licensed under CC-BY-NC-4.0.

Place, publisher, year, edition, pages
European Language Resources Association (ELRA), 2022
Keywords
Learning systems, Cross-lingual, Generalisation, Model knowledge, Model representation, Pre-training, Semantics knowledge, Source language, Semantics
National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-62612 (URN)2-s2.0-85144427655 (Scopus ID)9791095546726 (ISBN)
Conference
13th International Conference on Language Resources and Evaluation Conference, LREC 2022, 20 June 2022 through 25 June 2022
Note

 Funding details: VINNOVA, 2019-02996; Funding text 1: This work is supported by the Swedish innovation agency (Vinnova) under contract 2019-02996. 

Available from: 2023-01-24 Created: 2023-01-24 Last updated: 2023-01-24Bibliographically approved
Dwibedi, C., Mellergård, E., Gyllensten, A. C., Nilsson, K., Axelsson, A. S., Bäckman, M., . . . Rosengren, A. H. (2022). Effect of self-managed lifestyle treatment on glycemic control in patients with type 2 diabetes. npj Digital Medicine, 5(1), Article ID 60.
Open this publication in new window or tab >>Effect of self-managed lifestyle treatment on glycemic control in patients with type 2 diabetes
Show others...
2022 (English)In: npj Digital Medicine, ISSN 2398-6352, Vol. 5, no 1, article id 60Article in journal (Refereed) Published
Abstract [en]

The lack of effective, scalable solutions for lifestyle treatment is a global clinical problem, causing severe morbidity and mortality. We developed a method for lifestyle treatment that promotes self-reflection and iterative behavioral change, provided as a digital tool, and evaluated its effect in 370 patients with type 2 diabetes (ClinicalTrials.gov identifier: NCT04691973). Users of the tool had reduced blood glucose, both compared with randomized and matched controls (involving 158 and 204 users, respectively), as well as improved systolic blood pressure, body weight and insulin resistance. The improvement was sustained during the entire follow-up (average 730 days). A pathophysiological subgroup of obese insulin-resistant individuals had a pronounced glycemic response, enabling identification of those who would benefit in particular from lifestyle treatment. Natural language processing showed that the metabolic improvement was coupled with the self-reflective element of the tool. The treatment is cost-saving because of improved risk factor control for cardiovascular complications. The findings open an avenue for self-managed lifestyle treatment with long-term metabolic efficacy that is cost-saving and can reach large numbers of people. © 2022, The Author(s).

Place, publisher, year, edition, pages
Nature Research, 2022
Keywords
Blood, Blood pressure, Digital devices, Insulin, Iterative methods, Metabolism, Natural language processing systems, Behavioral changes, Clinical problems, Cost saving, Digital tools, Glycemic control, IS costs, Scalable solution, Self reflection, Self-managed, Type-2 diabetes, Patient treatment
National Category
Clinical Medicine
Identifiers
urn:nbn:se:ri:diva-60532 (URN)10.1038/s41746-022-00606-9 (DOI)2-s2.0-85129954140 (Scopus ID)
Note

 Funding details: Knut och Alice Wallenbergs Stiftelse; Funding text 1: Supported by grants from the Knut and Alice Wallenberg Foundation. We thank Svetlana Johansson, Paul Tyler, Anna-Maria Veljanovska Ramsay, Jasmina Kravic, as well as Louise Qvist, Maria Fälemark, Helene Ferm and Jessica Hedin for managing study visits. We also thank the patients and colleagues who participated in the development and evaluation of the tool.

Available from: 2022-10-19 Created: 2022-10-19 Last updated: 2022-10-19Bibliographically approved
Carlsson, F., Öhman, J., Liu, F., Verlinden, S., Nirve, J. & Sahlgren, M. (2022). Fine-Grained Controllable Text Generation Using Non-Residual Prompting. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): . Paper presented at 60th Annual Meeting of the Association for Computational Linguistics (pp. 6837-6857).
Open this publication in new window or tab >>Fine-Grained Controllable Text Generation Using Non-Residual Prompting
Show others...
2022 (English)In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, p. 6837-6857Conference paper, Published paper (Refereed)
Abstract [en]

The introduction of immensely large Causal Language Models (CLMs) has rejuvenated the interest in open-ended text generation. However, controlling the generative process for these Transformer-based models is at large an unsolved problem. Earlier work has explored either plug-and-play decoding strategies, or more powerful but blunt approaches such as prompting. There hence currently exists a trade-off between fine-grained control, and the capability for more expressive high-level instructions. To alleviate this trade-off, we propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. We propose a resource-efficient method for converting a pre-trained CLM into this architecture, and demonstrate its potential on various experiments, including the novel task of contextualized word inclusion. Our method provides strong results on multiple experimental settings, proving itself to be both expressive and versatile.

National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:ri:diva-59296 (URN)978-1-955917-21-6 (ISBN)
Conference
60th Annual Meeting of the Association for Computational Linguistics
Available from: 2022-05-30 Created: 2022-05-30 Last updated: 2023-05-16Bibliographically approved
Sahlgren, M., Carlsson, F., Olsson, F. & Börjeson, L. (2021). It’s Basically the Same Language Anyway: the Case for a Nordic Language Model. In: Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa): . Paper presented at 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), May 31-2 June 2021, Rykjavik, Iceland (pp. 367-372).
Open this publication in new window or tab >>It’s Basically the Same Language Anyway: the Case for a Nordic Language Model
2021 (English)In: Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), 2021, p. 367-372Conference paper, Published paper (Refereed)
Abstract [en]

When is it beneficial for a research community to organize a broader collaborative effort on a topic, and when should we instead promote individual efforts? In this opinion piece, we argue that we are at a stage in the development of large-scale language models where a collaborative effort is desirable, despite the fact that the preconditions for making individual contributions have never been better. We consider a number of arguments for collaboratively developing a large-scale Nordic language model, include environmental considerations, cost, data availability, language typology, cultural similarity, and transparency. Our primary goal is to raise awareness and foster a discussion about our potential impact and responsibility as NLP community.

National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-59817 (URN)
Conference
23rd Nordic Conference on Computational Linguistics (NoDaLiDa), May 31-2 June 2021, Rykjavik, Iceland
Available from: 2022-07-28 Created: 2022-07-28 Last updated: 2023-05-16
Gogoulou, E., Boman, M., Ben Abdesslem, F., Isacsson, N., Kaldo, V. & Sahlgren, M. (2021). Predicting treatment outcome from patient texts: The case of internet-based cognitive behavioural therapy. In: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference: . Paper presented at 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021, 19 April 2021 through 23 April 2021 (pp. 575-580). Association for Computational Linguistics (ACL)
Open this publication in new window or tab >>Predicting treatment outcome from patient texts: The case of internet-based cognitive behavioural therapy
Show others...
2021 (English)In: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics (ACL) , 2021, p. 575-580Conference paper, Published paper (Refereed)
Abstract [en]

We investigate the feasibility of applying standard text categorisation methods to patient text in order to predict treatment outcome in Internet-based cognitive behavioural therapy. The data set is unique in its detail and size for regular care for depression, social anxiety, and panic disorder. Our results indicate that there is a signal in the depression data, albeit a weak one. We also perform terminological and sentiment analysis, which confirm those results. 

Place, publisher, year, edition, pages
Association for Computational Linguistics (ACL), 2021
Keywords
Computational linguistics, Sentiment analysis, Data set, Internet based, Social anxieties, Treatment outcomes, Patient treatment
National Category
Applied Psychology
Identifiers
urn:nbn:se:ri:diva-53529 (URN)2-s2.0-85107290691 (Scopus ID)9781954085022 (ISBN)
Conference
16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021, 19 April 2021 through 23 April 2021
Available from: 2021-06-17 Created: 2021-06-17 Last updated: 2021-06-17Bibliographically approved
Carlsson, F., Gogoulou, E., Ylipää, E., Cuba Gyllensten, A. & Sahlgren, M. (2021). Semantic Re-tuning with Contrastive Tension. In: : . Paper presented at International Conference on Learning Representations, 2021.
Open this publication in new window or tab >>Semantic Re-tuning with Contrastive Tension
Show others...
2021 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Extracting semantically useful natural language sentence representations frompre-trained deep neural networks such as Transformers remains a challenge. Wefirst demonstrate that pre-training objectives impose a significant task bias ontothe final layers of models, with a layer-wise survey of the Semantic Textual Similarity (STS) correlations for multiple common Transformer language models. Wethen propose a new self-supervised method called Contrastive Tension (CT) tocounter such biases. CT frames the training objective as a noise-contrastive taskbetween the final layer representations of two independent models, in turn makingthe final layer representations suitable for feature extraction. Results from multiple common unsupervised and supervised STS tasks indicate that CT outperformsprevious State Of The Art (SOTA), and when combining CT with supervised datawe improve upon previous SOTA results with large margins.

National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-59816 (URN)
Conference
International Conference on Learning Representations, 2021
Available from: 2022-07-28 Created: 2022-07-28 Last updated: 2023-05-16
Sahlgren, M. & Carlsson, F. (2021). The Singleton Fallacy: Why Current Critiques of Language Models Miss the Point. Frontiers in Artificial Intelligence, 4, Article ID 682578.
Open this publication in new window or tab >>The Singleton Fallacy: Why Current Critiques of Language Models Miss the Point
2021 (English)In: Frontiers in Artificial Intelligence, E-ISSN 2624-8212, Vol. 4, article id 682578Article in journal (Refereed) Published
Abstract [en]

This paper discusses the current critique against neural network-based Natural Language Understanding solutions known as language models. We argue that much of the current debate revolves around an argumentation error that we refer to as the singleton fallacy: the assumption that a concept (in this case, language, meaning, and understanding) refers to a single and uniform phenomenon, which in the current debate is assumed to be unobtainable by (current) language models. By contrast, we argue that positing some form of (mental) “unobtanium” as definiens for understanding inevitably leads to a dualistic position, and that such a position is precisely the original motivation for developing distributional methods in computational linguistics. As such, we argue that language models present a theoretically (and practically) sound approach that is our current best bet for computers to achieve language understanding. This understanding must however be understood as a computational means to an end.

Place, publisher, year, edition, pages
Frontiers Media S.A., 2021
Keywords
language models, meaning, natural language understanding, neural networks, representation learning
National Category
Philosophy
Identifiers
urn:nbn:se:ri:diva-56930 (URN)10.3389/frai.2021.682578 (DOI)2-s2.0-85117916535 (Scopus ID)
Available from: 2021-11-22 Created: 2021-11-22 Last updated: 2023-05-16Bibliographically approved
Ghoorchian, K. & Sahlgren, M. (2020). GDTM: Graph-based Dynamic Topic Models. Progress in Artificial Intelligence, 9, 195-207
Open this publication in new window or tab >>GDTM: Graph-based Dynamic Topic Models
2020 (English)In: Progress in Artificial Intelligence, ISSN 2192-6352, Vol. 9, p. 195-207Article in journal (Refereed) Published
Abstract [en]

Dynamic Topic Modeling (DTM) is the ultimate solution for extracting topics from short texts generated in Online Social Networks (OSNs) like Twitter. It requires to be scalable and to be able to account for sparsity and dynamicity of short texts. Current solutions combine probabilistic mixture models like Dirichlet Multinomial or Pitman-Yor Process with approximate inference approaches like Gibbs Sampling and Stochastic Variational Inference to, respectively, account for dynamicity and scalability of DTM. However, these methods basically rely on weak probabilistic language models, which do not account for sparsity in short texts. In addition, their inference is based on iterative optimizations, which have scalability issues when it comes to DTM. We present GDTM, a single-pass graph-based DTM algorithm, to solve the problem. GDTM combines a context-rich and incremental feature representation method with graph partitioning to address scalability and dynamicity and uses a rich language model to account for sparsity. We run multiple experiments over a large-scale Twitter dataset to analyze the accuracy and scalability of GDTM and compare the results with four state-of-the-art models. In result, GDTM outperforms the best model by 11 % on accuracy and performs by an order of magnitude faster while creating four times better topic quality over standard evaluation metrics. © 2020, The Author(s).

Place, publisher, year, edition, pages
Springer, 2020
Keywords
Dimensionality reduction, Distributional semantics, Graph partitioning, Language modeling, Topic modeling
National Category
Natural Sciences
Identifiers
urn:nbn:se:ri:diva-45108 (URN)10.1007/s13748-020-00206-2 (DOI)2-s2.0-85085024680 (Scopus ID)
Available from: 2020-06-18 Created: 2020-06-18 Last updated: 2021-06-08Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0001-5100-0535

Search in DiVA

Show all publications