Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Developing AI Applications for the HPC-Cloud Continuum with ColonyOS
RISE Research Institutes of Sweden, Digital Systems, Data Science.ORCID iD: 0009-0001-1674-2506
RISE Research Institutes of Sweden, Digital Systems, Data Science.ORCID iD: 0000-0002-1655-3676
2024 (English)In: 2024 23rd International Symposium on Parallel and Distributed Computing, ISPDC 2024, Institute of Electrical and Electronics Engineers Inc. , 2024Conference paper, Published paper (Refereed)
Abstract [en]

Artificial Intelligence (AI) and machine learning have seen significant growth in recent years, leading to an increased demand for computational resources. To meet this demand and to boost Europe’s competitive edge, the European Commission has built several supercomputers across the continent. However, these traditional supercomputers often lack modern APIs and automation tools necessary for AI development. This paper outlines how High-Performance Computing (HPC) systems and cloud platforms can be seamlessly integrated using ColonyOS, an open-source meta-operating system designed to connect and integrate diverse computing environments into a cohesive compute continuum. ColonyOS enables development of AI workflows that are portable across HPC systems and cloud environments, including Kubernetes, Docker, and Slurm. This integration makes it possible to develop automation workflows, such as training AI models on HPC systems and automatically deploy trained models to cloud platforms for inference. The paper details the architecture of ColonyOS and how it can be used to build AI applications that can run in an HPC-Cloud Continuum. This will be exemplified through a satellite image segmentation case study, showcasing the benefits of combining the Leonardo EuroHPC supercomputer with a Kubernetes cluster. Ultimately, ColonyOS paves the way for hyper-distributed AI applications that can seamlessly utilize both cloud and HPC systems, including existing EuroHPC supercomputers.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc. , 2024.
Keywords [en]
Cluster computing; Open source software; Supercomputers; Artificial intelligence learning; Cloud platforms; Competitive edges; Computational resources; Computing clouds; European Commission; High performance computing systems; Machine-learning; Performance computing; Work-flows; Cloud platforms
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
URN: urn:nbn:se:ri:diva-76096DOI: 10.1109/ISPDC62236.2024.10705401Scopus ID: 2-s2.0-85207852696OAI: oai:DiVA.org:ri-76096DiVA, id: diva2:1932784
Conference
23rd International Symposium on Parallel and Distributed Computing, ISPDC 2024. Chur. 8 July 2024 through 10 July 2024
Note

This work was funded by the Vinnova funding agencyand the EuroHPC Joint Undertaking under EuroCC National Competence Center Sweden (ENCCS).

Available from: 2025-01-29 Created: 2025-01-29 Last updated: 2025-09-23Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Kristiansson, JohanWikfeldt, Thor kjartanthorwi@ri.se

Search in DiVA

By author/editor
Kristiansson, JohanWikfeldt, Thor kjartanthorwi@ri.se
By organisation
Data Science
Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 62 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf