Change search
Link to record
Permanent link

Direct link
Kleyko, Denis, Ph.D.ORCID iD iconorcid.org/0000-0002-6032-6155
Publications (10 of 26) Show all publications
Schlegel, K., Kleyko, D., Brinkmann, B. H., Nurse, E. S., Gayler, R. W. & Neubert, P. (2024). Lessons from a challenge on forecasting epileptic seizures from non-cerebral signals. Nature Machine Intelligence, 6(2), 243-244
Open this publication in new window or tab >>Lessons from a challenge on forecasting epileptic seizures from non-cerebral signals
Show others...
2024 (English)In: Nature Machine Intelligence, ISSN 2522-5839, Vol. 6, no 2, p. 243-244Article in journal (Refereed) Published
Abstract [en]

The “My Seizure Gauge” competition explored the challenge of forecasting epileptic seizures using non-invasive wearable devices without an electroencephalogram. The organizers and the winning team reflect on their experiences. 

Place, publisher, year, edition, pages
Nature Research, 2024
Keywords
Epileptic seizures; Wearable devices; Neurophysiology
National Category
Clinical Medicine
Identifiers
urn:nbn:se:ri:diva-73015 (URN)10.1038/s42256-024-00799-6 (DOI)2-s2.0-85185110710 (Scopus ID)
Available from: 2024-04-17 Created: 2024-04-17 Last updated: 2024-04-17Bibliographically approved
Kleyko, D., Rachkovskij, D., Osipov, E. & Rahimi, A. (2023). A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges. ACM Computing Surveys, 55(9), Article ID 3558000.
Open this publication in new window or tab >>A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
2023 (English)In: ACM Computing Surveys, ISSN 0360-0300, E-ISSN 1557-7341, Vol. 55, no 9, article id 3558000Article in journal (Refereed) Published
Abstract [en]

This is Part II of the two-part comprehensive survey devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Holographic Reduced Representations [321, 326] is an influential HDC/VSA model that is well known in the machine learning domain and often used to refer to the whole family. However, for the sake of consistency, we use HDC/VSA to refer to the field.Part I of this survey [222] covered foundational aspects of the field, such as the historical context leading to the development of HDC/VSA, key elements of any HDC/VSA model, known HDC/VSA models, and the transformation of input data of various types into high-dimensional vectors suitable for HDC/VSA. This second part surveys existing applications, the role of HDC/VSA in cognitive computing and architectures, as well as directions for future work. Most of the applications lie within the Machine Learning/Artificial Intelligence domain; however, we also cover other applications to provide a complete picture. The survey is written to be useful for both newcomers and practitioners. 

Place, publisher, year, edition, pages
Association for Computing Machinery, 2023
Keywords
analogical reasoning, applications, Artificial intelligence, binary spatter codes, cognitive architectures, cognitive computing, distributed representations, geometric analogue of holographic reduced representations, holographic reduced representations, hyperdimensional computing, machine learning, matrix binding of additive terms, modular composite representations, multiply-add-permute, sparse binary distributed representations, sparse block codes, tensor product representations, vector symbolic architectures, Architecture, Codes (symbols), Computer architecture, Holography, Metadata, Binary spatter code, Composite representations, Distributed representation, Geometric analog of holographic reduced representation, Machine-learning, Matrix binding, Matrix binding of additive term, Modular composite representation, Modulars, Multiply-add, Product representation, Sparse binary distributed representation, Sparse block code, Tensor product representation, Tensor products, Vector symbolic architecture, Vectors
National Category
Computer Sciences
Identifiers
urn:nbn:se:ri:diva-64721 (URN)10.1145/3558000 (DOI)2-s2.0-85147845869 (Scopus ID)
Note

Funding details: Air Force Office of Scientific Research, AFOSR, FA9550-19-1-0241; Funding details: Intel Corporation; Funding details: H2020 Marie Skłodowska-Curie Actions, MSCA, 839179; Funding details: Stiftelsen för Strategisk Forskning, SSF, UKR22-0024; Funding details: National Academy of Sciences of Ukraine, NASU, 0117U002286, 0120U000122, 0121U000016, 0122U002151; Funding details: Ministry of Education and Science of Ukraine, MESU, 0121U000228, 0122U000818; Funding text 1: The work of DK was supported by the European Union’s Horizon 2020 Programme under the Marie Skłodowska-Curie Individual Fellowship Grant (839179). The work of DK was also supported in part by AFOSR FA9550-19-1-0241 and Intel’s THWAI program. The work of DAR was supported in part by the National Academy of Sciences of Ukraine (grant no. 0120U000122, 0121U000016, 0122U002151, and 0117U002286), the Ministry of Education and Science of Ukraine (grant no. 0121U000228 and 0122U000818), and the Swedish Foundation for Strategic Research (SSF, grant no. UKR22-0024).; Funding text 2: The work of DK was supported by the European Union’s Horizon 2020 Programme under the Marie SkÅ odowska-Curie Individual Fellowship Grant (839179). The work of DK was also supported in part by AFOSR FA9550-19-1-0241 and Intel’s THWAI program. The work of DAR was supported in part by the National Academy of Sciences of Ukraine (grant no. 0120U000122, 0121U000016, 0122U002151, and 0117U002286), the Ministry of Education and Science of Ukraine (grant no. 0121U000228 and 0122U000818), and the Swedish Foundation for Strategic Research (SSF, grant no. UKR22-0024).

Available from: 2023-05-15 Created: 2023-05-15 Last updated: 2023-12-12Bibliographically approved
Srivastava, A., Kleyko, D. & Wu, Z. (2023). Beyond the Imitation Game: Quantifying and extrapolatingthe capabilities of language models. Transactions on Machine Learning Research (5)
Open this publication in new window or tab >>Beyond the Imitation Game: Quantifying and extrapolatingthe capabilities of language models
2023 (English)In: Transactions on Machine Learning Research, E-ISSN 2835-8856, no 5Article in journal (Refereed) Published
Abstract [en]

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG- bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood develop- ment, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google- internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:ri:diva-67825 (URN)10.48550/arXiv.2206.04615 (DOI)
Available from: 2023-11-16 Created: 2023-11-16 Last updated: 2024-01-08Bibliographically approved
Kleyko, D., Bybee, C., Huang, P.-C., Kymn, C., Olshausen, B., Frady, E. P. & Sommer, F. (2023). Efficient Decoding of Compositional Structure in Holistic Representations. Neural Computation, 35(7), 1159-1186
Open this publication in new window or tab >>Efficient Decoding of Compositional Structure in Holistic Representations
Show others...
2023 (English)In: Neural Computation, ISSN 0899-7667, E-ISSN 1530-888X, Vol. 35, no 7, p. 1159-1186Article in journal (Refereed) Published
Abstract [en]

We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks. 

Place, publisher, year, edition, pages
MIT Press Journals, 2023
Keywords
Architecture, Computer architecture, Codebooks, Compositional structure, Decoding techniques, Distributed representation, External noise, Four-group, Information rates, Novel techniques, Reduced precision, Storage elements, article, information retrieval, noise, Decoding
National Category
Telecommunications
Identifiers
urn:nbn:se:ri:diva-65998 (URN)10.1162/neco_a_01590 (DOI)2-s2.0-85163619213 (Scopus ID)
Note

We thank Spencer Kent for sharing with us the implementation of the FISTAalgorithm. The work of F.T.S., B.A.O., C.B., and D.K. was supported in partby Intel’s THWAI program. The work of B.A.O. and D.K. was also supported in part by AFOSR FA9550-19-1-0241. D.K. has received funding fromthe European Union’s Horizon 2020 research and innovation program. under Marie Sklodowska-Curie grant agreement 839179. The work of C.J.K.was supported by the U.S. Department of Defense through the NationalDefense Science and Engineering Graduate Fellowship Program. F.T.S. wassupported by Intel and NIH R01-EB026955.

Available from: 2023-08-22 Created: 2023-08-22 Last updated: 2023-12-12Bibliographically approved
Bybee, C., Kleyko, D., Nikonov, D. E., Khosrowshahi, A., Olshausen, B. A. & Sommer, F. T. (2023). Efficient optimization with higher-order ising machines. Nature Communications, 14, Article ID 6033.
Open this publication in new window or tab >>Efficient optimization with higher-order ising machines
Show others...
2023 (English)In: Nature Communications, E-ISSN 2041-1723, Vol. 14, article id 6033Article in journal (Refereed) Published
Abstract [en]

A prominent approach to solving combinatorial optimization problems on parallel hardware is Ising machines, i.e., hardware implementations of networks of interacting binary spin variables. Most Ising machines leverage second-order interactions although important classes of optimization problems, such as satisfiability problems, map more seamlessly to Ising networks with higher-order interactions. Here, we demonstrate that higher-order Ising machines can solve satisfiability problems more resource-efficiently in terms of the number of spin variables and their connections when compared to traditional second-order Ising machines. Further, our results show on a benchmark dataset of Boolean k-satisfiability problems that higher-order Ising machines implemented with coupled oscillators rapidly find solutions that are better than second-order Ising machines, thus, improving the current state-of-the-art for Ising machines. 

Place, publisher, year, edition, pages
Nature Research, 2023
Keywords
metal oxide, benchmarking; data set; hardware; optimization, Article; data availability; decomposition; machine; model; process optimization; satisfaction; simulated annealing
National Category
Physical Sciences
Identifiers
urn:nbn:se:ri:diva-67762 (URN)10.1038/s41467-023-41214-9 (DOI)2-s2.0-85172783609 (Scopus ID)
Note

C.B. acknowledges support from the National Science Foundation (NSF) through an NSF Graduate Research Fellowships Program (GRFP) fellowship (DGE 1752814) and an Intel Corporation research grant. D.K. acknowledges support from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 839179. F.T.S. was supported by NSF Grant IIS1718991 and NIH Grant 1R01EB026955. B.A.O. was supported by NSF EAGER grant 2147640.

Available from: 2023-11-29 Created: 2023-11-29 Last updated: 2023-12-12Bibliographically approved
Dhole, K., Kleyko, D. & Zhang, Y. (2023). NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation. NEJLT Northern European Journal of Language Technology, 9(1), 1-41
Open this publication in new window or tab >>NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
2023 (English)In: NEJLT Northern European Journal of Language Technology, ISSN 2000-1533, Vol. 9, no 1, p. 1-41Article in journal (Refereed) Published
Abstract [en]

Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training datafor natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based naturallanguage (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters(data splits according to specific features). We describe the framework and an initial set of117transformations and23filters for avariety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental humanmistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguousto humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popularlanguage models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases.The infrastructure, datacards, and robustness evaluation results are publicly available onGitHubfor the benefit of researchersworking on paraphrase generation, robustness analysis, and low-resource NLP.

National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:ri:diva-67824 (URN)10.3384/nejlt.2000-1533.2023.4725 (DOI)
Available from: 2023-11-16 Created: 2023-11-16 Last updated: 2023-12-12Bibliographically approved
Teeters, J., Kleyko, D., Kanerva, P. & Olshausen, B. (2023). On separating long- and short-term memories in hyperdimensional computing. Frontiers in Neuroscience, 16, Article ID 867568.
Open this publication in new window or tab >>On separating long- and short-term memories in hyperdimensional computing
2023 (English)In: Frontiers in Neuroscience, ISSN 1662-4548, E-ISSN 1662-453X, Vol. 16, article id 867568Article in journal (Refereed) Published
Abstract [en]

Operations on high-dimensional, fixed-width vectors can be used to distribute information from several vectors over a single vector of the same width. For example, a set of key-value pairs can be encoded into a single vector with multiplication and addition of the corresponding key and value vectors: the keys are bound to their values with component-wise multiplication, and the key-value pairs are combined into a single superposition vector with component-wise addition. The superposition vector is, thus, a memory which can then be queried for the value of any of the keys, but the result of the query is approximate. The exact vector is retrieved from a codebook (a.k.a. item memory), which contains vectors defined in the system. To perform these operations, the item memory vectors and the superposition vector must be the same width. Increasing the capacity of the memory requires increasing the width of the superposition and item memory vectors. In this article, we demonstrate that in a regime where many (e.g., 1,000 or more) key-value pairs are stored, an associative memory which maps key vectors to value vectors requires less memory and less computing to obtain the same reliability of storage as a superposition vector. These advantages are obtained because the number of storage locations in an associate memory can be increased without increasing the width of the vectors in the item memory. An associative memory would not replace a superposition vector as a medium of storage, but could augment it, because data recalled from an associative memory could be used in algorithms that use a superposition vector. This would be analogous to how human working memory (which stores about seven items) uses information recalled from long-term memory (which is much larger than the working memory). We demonstrate the advantages of an associative memory experimentally using the storage of large finite-state automata, which could model the storage and recall of state-dependent behavior by brains. 

Place, publisher, year, edition, pages
Frontiers Media S.A., 2023
Keywords
associative memory, holographic reduced representation, hyperdimensional computing, long-term memory, short-term memory, sparse distributed memory, vector symbolic architectures, working memory, algorithm, article, brain, finite state machine, human, human experiment, long term memory, memory, recall, reliability, short term memory
National Category
Natural Sciences
Identifiers
urn:nbn:se:ri:diva-64723 (URN)10.3389/fnins.2022.867568 (DOI)2-s2.0-85146846574 (Scopus ID)
Note

 Funding details: FA9550-19-1-0241; Funding details: H2020 Marie Skłodowska-Curie Actions, MSCA, 839179; Funding details: Horizon 2020; Funding text 1: This research was supported by Air Force Office of Scientific Research Program on Cognitive and Computational Neuroscience, FA9550-19-1-0241. DK has received funding from the European Union's Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement no. 839179. Publication made possible in part by support from the Berkeley Research Impact Initiative (BRII) sponsored by the UC Berkeley Library.

Available from: 2023-05-15 Created: 2023-05-15 Last updated: 2023-12-12Bibliographically approved
Kleyko, D., Rosato, A., Frady, E. P., Panella, M. & Sommer, F. T. (2023). Perceptron Theory Can Predict the Accuracy of Neural Networks. IEEE Transactions on Neural Networks and Learning Systems
Open this publication in new window or tab >>Perceptron Theory Can Predict the Accuracy of Neural Networks
Show others...
2023 (English)In: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388Article in journal (Refereed) Epub ahead of print
Abstract [en]

Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. Furthermore, the theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. Moreover, the perceptron theory compares favorably to other methods that do not rely on training an estimator model.

National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:ri:diva-67822 (URN)10.1109/TNNLS.2023.3237381 (DOI)
Available from: 2023-11-22 Created: 2023-11-22 Last updated: 2023-12-12Bibliographically approved
Heddes, M., Nunes, I., Verges, P., Kleyko, D., Abraham, D., Givargis, T., . . . Veidenbaum, A. (2023). Torchhd: An Open Source Python Library to SupportResearch on Hyperdimensional Computing andVector Symbolic Architectures. Journal of Machine Learning Research, 24, 1-10
Open this publication in new window or tab >>Torchhd: An Open Source Python Library to SupportResearch on Hyperdimensional Computing andVector Symbolic Architectures
Show others...
2023 (English)In: Journal of Machine Learning Research, Vol. 24, p. 1-10Article in journal (Refereed) Published
Abstract [en]

Hyperdimensional computing (HD), also known as vector symbolic architectures (VSA), is a framework for computing with distributed representations by exploiting properties of random high-dimensional vector spaces. The commitment of the scientific community to aggregate and disseminate research in this particularly multidisciplinary area has been fundamental for its advancement. Joining these efforts, we present Torchhd, a highperformance open source Python library for HD/VSA. Torchhd seeks to make HD/VSA more accessible and serves as an efficient foundation for further research and application development. The easy-to-use library builds on top of PyTorch and features state-of-the art HD/VSA functionality, clear documentation, and implementation examples from wellknown publications. Comparing publicly available code with their corresponding Torchhd implementation shows that experiments can run up to 100× faster. Torchhd is available at: https://github.com/hyperdimensional-computing/torchhd.

National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:ri:diva-67826 (URN)
Available from: 2023-11-16 Created: 2023-11-16 Last updated: 2023-12-12Bibliographically approved
Frady, E. P., Kleyko, D. & Sommer, F. T. (2023). Variable Binding for Sparse Distributed Representations: Theory and Applications. IEEE Transactions on Neural Networks and Learning Systems, 34(5), 2191-2204
Open this publication in new window or tab >>Variable Binding for Sparse Distributed Representations: Theory and Applications
2023 (English)In: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 34, no 5, p. 2191-2204Article in journal (Refereed) Published
Abstract [en]

Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the context of neuroscience and neural networks. 

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc., 2023
Keywords
Brain; Compressed sensing; Convolution; Data structures; Matrix algebra; Network architecture; Neural networks; Neurons; Tensors; Trees (mathematics), Cognition; Cognitive reasoning; Compound; Compressed sensing; Compressed-Sensing; Distributed representation; Sparse block code; Sparse distributed representation; Sparse matrices; Tensor product variable binding; Tensor products; Variable binding; Vector symbolic architecture ., Vectors, artificial neural network; brain; cognition; nerve cell; physiology; problem solving, Brain; Cognition; Neural Networks, Computer; Neurons; Problem Solving
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:ri:diva-67764 (URN)10.1109/TNNLS.2021.3105949 (DOI)2-s2.0-85114716481 (Scopus ID)
Available from: 2023-11-22 Created: 2023-11-22 Last updated: 2023-12-12Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-6032-6155

Search in DiVA

Show all publications