Change search
Link to record
Permanent link

Direct link
Kleyko, Denis, Ph.D.ORCID iD iconorcid.org/0000-0002-6032-6155
Publications (10 of 26) Show all publications
Schlegel, K., Kleyko, D., Brinkmann, B. H., Nurse, E. S., Gayler, R. W. & Neubert, P. (2024). Lessons from a challenge on forecasting epileptic seizures from non-cerebral signals. Nature Machine Intelligence, 6(2), 243-244
Open this publication in new window or tab >>Lessons from a challenge on forecasting epileptic seizures from non-cerebral signals
Show others...
2024 (English)In: Nature Machine Intelligence, ISSN 2522-5839, Vol. 6, no 2, p. 243-244Article in journal (Refereed) Published
Abstract [en]

The “My Seizure Gauge” competition explored the challenge of forecasting epileptic seizures using non-invasive wearable devices without an electroencephalogram. The organizers and the winning team reflect on their experiences. 

Place, publisher, year, edition, pages
Nature Research, 2024
Keywords
Epileptic seizures; Wearable devices; Neurophysiology
National Category
Clinical Medicine
Identifiers
urn:nbn:se:ri:diva-73015 (URN)10.1038/s42256-024-00799-6 (DOI)2-s2.0-85185110710 (Scopus ID)
Available from: 2024-04-17 Created: 2024-04-17 Last updated: 2024-04-17Bibliographically approved
Kleyko, D., Rachkovskij, D., Osipov, E. & Rahimi, A. (2023). A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges. ACM Computing Surveys, 55(9), Article ID 3558000.
Open this publication in new window or tab >>A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
2023 (English)In: ACM Computing Surveys, ISSN 0360-0300, E-ISSN 1557-7341, Vol. 55, no 9, article id 3558000Article in journal (Refereed) Published
Abstract [en]

This is Part II of the two-part comprehensive survey devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Holographic Reduced Representations [321, 326] is an influential HDC/VSA model that is well known in the machine learning domain and often used to refer to the whole family. However, for the sake of consistency, we use HDC/VSA to refer to the field.Part I of this survey [222] covered foundational aspects of the field, such as the historical context leading to the development of HDC/VSA, key elements of any HDC/VSA model, known HDC/VSA models, and the transformation of input data of various types into high-dimensional vectors suitable for HDC/VSA. This second part surveys existing applications, the role of HDC/VSA in cognitive computing and architectures, as well as directions for future work. Most of the applications lie within the Machine Learning/Artificial Intelligence domain; however, we also cover other applications to provide a complete picture. The survey is written to be useful for both newcomers and practitioners. 

Place, publisher, year, edition, pages
Association for Computing Machinery, 2023
Keywords
analogical reasoning, applications, Artificial intelligence, binary spatter codes, cognitive architectures, cognitive computing, distributed representations, geometric analogue of holographic reduced representations, holographic reduced representations, hyperdimensional computing, machine learning, matrix binding of additive terms, modular composite representations, multiply-add-permute, sparse binary distributed representations, sparse block codes, tensor product representations, vector symbolic architectures, Architecture, Codes (symbols), Computer architecture, Holography, Metadata, Binary spatter code, Composite representations, Distributed representation, Geometric analog of holographic reduced representation, Machine-learning, Matrix binding, Matrix binding of additive term, Modular composite representation, Modulars, Multiply-add, Product representation, Sparse binary distributed representation, Sparse block code, Tensor product representation, Tensor products, Vector symbolic architecture, Vectors
National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-64721 (URN)10.1145/3558000 (DOI)2-s2.0-85147845869 (Scopus ID)
Note

Funding details: Air Force Office of Scientific Research, AFOSR, FA9550-19-1-0241; Funding details: Intel Corporation; Funding details: H2020 Marie Skłodowska-Curie Actions, MSCA, 839179; Funding details: Stiftelsen för Strategisk Forskning, SSF, UKR22-0024; Funding details: National Academy of Sciences of Ukraine, NASU, 0117U002286, 0120U000122, 0121U000016, 0122U002151; Funding details: Ministry of Education and Science of Ukraine, MESU, 0121U000228, 0122U000818; Funding text 1: The work of DK was supported by the European Union’s Horizon 2020 Programme under the Marie Skłodowska-Curie Individual Fellowship Grant (839179). The work of DK was also supported in part by AFOSR FA9550-19-1-0241 and Intel’s THWAI program. The work of DAR was supported in part by the National Academy of Sciences of Ukraine (grant no. 0120U000122, 0121U000016, 0122U002151, and 0117U002286), the Ministry of Education and Science of Ukraine (grant no. 0121U000228 and 0122U000818), and the Swedish Foundation for Strategic Research (SSF, grant no. UKR22-0024).; Funding text 2: The work of DK was supported by the European Union’s Horizon 2020 Programme under the Marie SkÅ odowska-Curie Individual Fellowship Grant (839179). The work of DK was also supported in part by AFOSR FA9550-19-1-0241 and Intel’s THWAI program. The work of DAR was supported in part by the National Academy of Sciences of Ukraine (grant no. 0120U000122, 0121U000016, 0122U002151, and 0117U002286), the Ministry of Education and Science of Ukraine (grant no. 0121U000228 and 0122U000818), and the Swedish Foundation for Strategic Research (SSF, grant no. UKR22-0024).

Available from: 2023-05-15 Created: 2023-05-15 Last updated: 2023-12-12Bibliographically approved
Srivastava, A., Kleyko, D. & Wu, Z. (2023). Beyond the Imitation Game: Quantifying and extrapolatingthe capabilities of language models. Transactions on Machine Learning Research (5)
Open this publication in new window or tab >>Beyond the Imitation Game: Quantifying and extrapolatingthe capabilities of language models
2023 (English)In: Transactions on Machine Learning Research, E-ISSN 2835-8856, no 5Article in journal (Refereed) Published
Abstract [en]

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG- bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood develop- ment, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google- internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:ri:diva-67825 (URN)10.48550/arXiv.2206.04615 (DOI)
Available from: 2023-11-16 Created: 2023-11-16 Last updated: 2024-01-08Bibliographically approved
Kleyko, D., Bybee, C., Huang, P.-C., Kymn, C., Olshausen, B., Frady, E. P. & Sommer, F. (2023). Efficient Decoding of Compositional Structure in Holistic Representations. Neural Computation, 35(7), 1159-1186
Open this publication in new window or tab >>Efficient Decoding of Compositional Structure in Holistic Representations
Show others...
2023 (English)In: Neural Computation, ISSN 0899-7667, E-ISSN 1530-888X, Vol. 35, no 7, p. 1159-1186Article in journal (Refereed) Published
Abstract [en]

We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks. 

Place, publisher, year, edition, pages
MIT Press Journals, 2023
Keywords
Architecture, Computer architecture, Codebooks, Compositional structure, Decoding techniques, Distributed representation, External noise, Four-group, Information rates, Novel techniques, Reduced precision, Storage elements, article, information retrieval, noise, Decoding
National Category
Telecommunications
Identifiers
urn:nbn:se:ri:diva-65998 (URN)10.1162/neco_a_01590 (DOI)2-s2.0-85163619213 (Scopus ID)
Note

We thank Spencer Kent for sharing with us the implementation of the FISTAalgorithm. The work of F.T.S., B.A.O., C.B., and D.K. was supported in partby Intel’s THWAI program. The work of B.A.O. and D.K. was also supported in part by AFOSR FA9550-19-1-0241. D.K. has received funding fromthe European Union’s Horizon 2020 research and innovation program. under Marie Sklodowska-Curie grant agreement 839179. The work of C.J.K.was supported by the U.S. Department of Defense through the NationalDefense Science and Engineering Graduate Fellowship Program. F.T.S. wassupported by Intel and NIH R01-EB026955.

Available from: 2023-08-22 Created: 2023-08-22 Last updated: 2023-12-12Bibliographically approved
Bybee, C., Kleyko, D., Nikonov, D. E., Khosrowshahi, A., Olshausen, B. A. & Sommer, F. T. (2023). Efficient optimization with higher-order ising machines. Nature Communications, 14, Article ID 6033.
Open this publication in new window or tab >>Efficient optimization with higher-order ising machines
Show others...
2023 (English)In: Nature Communications, E-ISSN 2041-1723, Vol. 14, article id 6033Article in journal (Refereed) Published
Abstract [en]

A prominent approach to solving combinatorial optimization problems on parallel hardware is Ising machines, i.e., hardware implementations of networks of interacting binary spin variables. Most Ising machines leverage second-order interactions although important classes of optimization problems, such as satisfiability problems, map more seamlessly to Ising networks with higher-order interactions. Here, we demonstrate that higher-order Ising machines can solve satisfiability problems more resource-efficiently in terms of the number of spin variables and their connections when compared to traditional second-order Ising machines. Further, our results show on a benchmark dataset of Boolean k-satisfiability problems that higher-order Ising machines implemented with coupled oscillators rapidly find solutions that are better than second-order Ising machines, thus, improving the current state-of-the-art for Ising machines. 

Place, publisher, year, edition, pages
Nature Research, 2023
Keywords
metal oxide, benchmarking; data set; hardware; optimization, Article; data availability; decomposition; machine; model; process optimization; satisfaction; simulated annealing
National Category
Physical Sciences
Identifiers
urn:nbn:se:ri:diva-67762 (URN)10.1038/s41467-023-41214-9 (DOI)2-s2.0-85172783609 (Scopus ID)
Note

C.B. acknowledges support from the National Science Foundation (NSF) through an NSF Graduate Research Fellowships Program (GRFP) fellowship (DGE 1752814) and an Intel Corporation research grant. D.K. acknowledges support from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 839179. F.T.S. was supported by NSF Grant IIS1718991 and NIH Grant 1R01EB026955. B.A.O. was supported by NSF EAGER grant 2147640.

Available from: 2023-11-29 Created: 2023-11-29 Last updated: 2023-12-12Bibliographically approved
Kleyko, D., Karunaratne, G., Rabaey, J. M., Sebastian, A. & Rahimi, A. (2023). Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks. IEEE Transactions on Neural Networks and Learning Systems, 34(12), 10993-10998
Open this publication in new window or tab >>Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks
Show others...
2023 (English)In: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 34, no 12, p. 10993-10998Article in journal (Refereed) Published
Abstract [en]

Memory-augmented neural networks enhance a neural network with an external key-value (KV) memory whose complexity is typically dominated by the number of support vectors in the key memory. We propose a generalized KV memory that decouples its dimension from the number of support vectors by introducing a free parameter that can arbitrarily add or remove redundancy to the key memory representation. In effect, it provides an additional degree of freedom to flexibly control the tradeoff between robustness and the resources required to store and compute the generalized KV memory. This is particularly useful for realizing the key memory on in-memory computing hardware where it exploits nonideal, but extremely efficient nonvolatile memory devices for dense storage and computation. Experimental results show that adapting this parameter on demand effectively mitigates up to 44% nonidealities, at equal accuracy and number of devices, without any need for neural network retraining.

National Category
Computer Systems
Identifiers
urn:nbn:se:ri:diva-63326 (URN)10.1109/tnnls.2022.3159445 (DOI)
Note

The work of Denis Kleyko was supported in part by the European Union’s Horizon 2020 Programme through the Marie Skłodowska-Curie Individual Fellowship under Grant 839179, in part by the Defense Advanced Research Projects Agency’s (DARPA’s) Artificial Intelligence Exploration (AIE) HyDDENN Project Program, and in part by the Air Force Office of Scientific Research (AFOSR) under Grant FA9550-19-1-0241. The work of Geethan Karunaratne and Abu Sebastianwas supported in part by the European Research Council (ERC) through the European Unions Horizon 2020 Research and Innovation Program under Grant 682675. The work of Jan M. Rabaey was supported in part by the DARPA’s AIE HyDDENN Project Program

Available from: 2023-01-30 Created: 2023-01-30 Last updated: 2024-06-11Bibliographically approved
Osipov, E., Kahawala, S., Haputhanthri, D., Kempitiya, T., Silva, D. D., Alahakoon, D. & Kleyko, D. (2023). Hyperseed: Unsupervised Learning With Vector Symbolic Architectures. IEEE Transactions on Neural Networks and Learning Systems, 12(12), Article ID e202300141.
Open this publication in new window or tab >>Hyperseed: Unsupervised Learning With Vector Symbolic Architectures
Show others...
2023 (English)In: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 12, no 12, article id e202300141Article in journal (Refereed) Published
Abstract [en]

Motivated by recent innovations in biologically inspired neuromorphic hardware, this article presents a novel unsupervised machine learning algorithm named Hyperseed that draws on the principles of vector symbolic architectures (VSAs) for fast learning of a topology preserving feature map of unlabeled data. It relies on two major operations of VSA, binding and bundling. The algorithmic part of Hyperseed is expressed within the Fourier holographic reduced representations (FHRR) model, which is specifically suited for implementation on spiking neuromorphic hardware. The two primary contributions of the Hyperseed algorithm are few-shot learning and a learning rule based on single vector operation. These properties are empirically evaluated on synthetic datasets and on illustrative benchmark use cases, IRIS classification, and a language identification task using the $n$ -gram statistics. The results of these experiments confirm the capabilities of Hyperseed and its applications in neuromorphic hardware.

National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-62510 (URN)10.1109/TNNLS.2022.3211274 (DOI)
Available from: 2023-01-23 Created: 2023-01-23 Last updated: 2024-06-11Bibliographically approved
Dhole, K., Kleyko, D. & Zhang, Y. (2023). NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation. NEJLT Northern European Journal of Language Technology, 9(1), 1-41
Open this publication in new window or tab >>NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
2023 (English)In: NEJLT Northern European Journal of Language Technology, ISSN 2000-1533, Vol. 9, no 1, p. 1-41Article in journal (Refereed) Published
Abstract [en]

Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training datafor natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based naturallanguage (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters(data splits according to specific features). We describe the framework and an initial set of117transformations and23filters for avariety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental humanmistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguousto humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popularlanguage models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases.The infrastructure, datacards, and robustness evaluation results are publicly available onGitHubfor the benefit of researchersworking on paraphrase generation, robustness analysis, and low-resource NLP.

National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:ri:diva-67824 (URN)10.3384/nejlt.2000-1533.2023.4725 (DOI)
Available from: 2023-11-16 Created: 2023-11-16 Last updated: 2023-12-12Bibliographically approved
Teeters, J., Kleyko, D., Kanerva, P. & Olshausen, B. (2023). On separating long- and short-term memories in hyperdimensional computing. Frontiers in Neuroscience, 16, Article ID 867568.
Open this publication in new window or tab >>On separating long- and short-term memories in hyperdimensional computing
2023 (English)In: Frontiers in Neuroscience, ISSN 1662-4548, E-ISSN 1662-453X, Vol. 16, article id 867568Article in journal (Refereed) Published
Abstract [en]

Operations on high-dimensional, fixed-width vectors can be used to distribute information from several vectors over a single vector of the same width. For example, a set of key-value pairs can be encoded into a single vector with multiplication and addition of the corresponding key and value vectors: the keys are bound to their values with component-wise multiplication, and the key-value pairs are combined into a single superposition vector with component-wise addition. The superposition vector is, thus, a memory which can then be queried for the value of any of the keys, but the result of the query is approximate. The exact vector is retrieved from a codebook (a.k.a. item memory), which contains vectors defined in the system. To perform these operations, the item memory vectors and the superposition vector must be the same width. Increasing the capacity of the memory requires increasing the width of the superposition and item memory vectors. In this article, we demonstrate that in a regime where many (e.g., 1,000 or more) key-value pairs are stored, an associative memory which maps key vectors to value vectors requires less memory and less computing to obtain the same reliability of storage as a superposition vector. These advantages are obtained because the number of storage locations in an associate memory can be increased without increasing the width of the vectors in the item memory. An associative memory would not replace a superposition vector as a medium of storage, but could augment it, because data recalled from an associative memory could be used in algorithms that use a superposition vector. This would be analogous to how human working memory (which stores about seven items) uses information recalled from long-term memory (which is much larger than the working memory). We demonstrate the advantages of an associative memory experimentally using the storage of large finite-state automata, which could model the storage and recall of state-dependent behavior by brains. 

Place, publisher, year, edition, pages
Frontiers Media S.A., 2023
Keywords
associative memory, holographic reduced representation, hyperdimensional computing, long-term memory, short-term memory, sparse distributed memory, vector symbolic architectures, working memory, algorithm, article, brain, finite state machine, human, human experiment, long term memory, memory, recall, reliability, short term memory
National Category
Natural Sciences
Identifiers
urn:nbn:se:ri:diva-64723 (URN)10.3389/fnins.2022.867568 (DOI)2-s2.0-85146846574 (Scopus ID)
Note

 Funding details: FA9550-19-1-0241; Funding details: H2020 Marie Skłodowska-Curie Actions, MSCA, 839179; Funding details: Horizon 2020; Funding text 1: This research was supported by Air Force Office of Scientific Research Program on Cognitive and Computational Neuroscience, FA9550-19-1-0241. DK has received funding from the European Union's Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement no. 839179. Publication made possible in part by support from the Berkeley Research Impact Initiative (BRII) sponsored by the UC Berkeley Library.

Available from: 2023-05-15 Created: 2023-05-15 Last updated: 2023-12-12Bibliographically approved
Kleyko, D., Rosato, A., Frady, E. P., Panella, M. & Sommer, F. T. (2023). Perceptron Theory Can Predict the Accuracy of Neural Networks. IEEE Transactions on Neural Networks and Learning Systems
Open this publication in new window or tab >>Perceptron Theory Can Predict the Accuracy of Neural Networks
Show others...
2023 (English)In: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388Article in journal (Refereed) Epub ahead of print
Abstract [en]

Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. Furthermore, the theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. Moreover, the perceptron theory compares favorably to other methods that do not rely on training an estimator model.

National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:ri:diva-67822 (URN)10.1109/TNNLS.2023.3237381 (DOI)
Available from: 2023-11-22 Created: 2023-11-22 Last updated: 2024-06-11Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-6032-6155

Search in DiVA

Show all publications