Ändra sökning
Avgränsa sökresultatet
1 - 26 av 26
RefereraExporteraLänk till träfflistan
Permanent länk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Träffar per sida
  • 5
  • 10
  • 20
  • 50
  • 100
  • 250
Sortering
  • Standard (Relevans)
  • Författare A-Ö
  • Författare Ö-A
  • Titel A-Ö
  • Titel Ö-A
  • Publikationstyp A-Ö
  • Publikationstyp Ö-A
  • Äldst först
  • Nyast först
  • Skapad (Äldst först)
  • Skapad (Nyast först)
  • Senast uppdaterad (Äldst först)
  • Senast uppdaterad (Nyast först)
  • Disputationsdatum (tidigaste först)
  • Disputationsdatum (senaste först)
  • Standard (Relevans)
  • Författare A-Ö
  • Författare Ö-A
  • Titel A-Ö
  • Titel Ö-A
  • Publikationstyp A-Ö
  • Publikationstyp Ö-A
  • Äldst först
  • Nyast först
  • Skapad (Äldst först)
  • Skapad (Nyast först)
  • Senast uppdaterad (Äldst först)
  • Senast uppdaterad (Nyast först)
  • Disputationsdatum (tidigaste först)
  • Disputationsdatum (senaste först)
Markera
Maxantalet träffar du kan exportera från sökgränssnittet är 250. Vid större uttag använd dig av utsökningar.
  • 1.
    Alonso, P.
    et al.
    Luleå University of Technology, Sweden.
    Shridhar, K.
    ETH Zürich, Switzerland.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap.
    Osipov, E.
    Luleå University of Technology, Sweden.
    Liwicki, M.
    Luleå University of Technology, Sweden.
    HyperEmbed: Tradeoffs between Resources and Performance in NLP Tasks with Hyperdimensional Computing Enabled Embedding of n-gram Statistics2021Ingår i: Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc. , 2021, Vol. 2021-JulyKonferensbidrag (Refereegranskat)
    Abstract [en]

    Recent advances in Deep Learning have led to a significant performance increase on several NLP tasks, however, the models become more and more computationally demanding. Therefore, this paper tackles the domain of computationally efficient algorithms for NLP tasks. In particular, it investigates distributed representations of n -gram statistics of texts. The representations are formed using hyperdimensional computing enabled embedding. These representations then serve as features, which are used as input to standard classifiers. We investigate the applicability of the embedding on one large and three small standard datasets for classification tasks using nine classifiers. The embedding achieved on par F_1 scores while decreasing the time and memory requirements by several times compared to the conventional n -gram statistics, e.g., for one of the classifiers on a small dataset, the memory reduction was 6.18 times; while train and test speed-ups were 4.62 and 3.84 times, respectively. For many classifiers on the large dataset, memory reduction was ca. 100 times and train and test speed-ups were over 100 times. Importantly, the usage of distributed representations formed via hyperdimensional computing allows dissecting strict dependency between the dimensionality of the representation and n-gram size, thus, opening a room for tradeoffs. 

  • 2.
    Bybee, Connor
    et al.
    University of California, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Nikonov, Dmitri E
    Intel, USA.
    Khosrowshahi, A.
    University of California, USA; Intel, USA.
    Olshausen, B. A.
    University of California, USA.
    Sommer, F. T.
    University of California, USA; Intel, USA.
    Efficient optimization with higher-order ising machines2023Ingår i: Nature Communications, E-ISSN 2041-1723, Vol. 14, artikel-id 6033Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    A prominent approach to solving combinatorial optimization problems on parallel hardware is Ising machines, i.e., hardware implementations of networks of interacting binary spin variables. Most Ising machines leverage second-order interactions although important classes of optimization problems, such as satisfiability problems, map more seamlessly to Ising networks with higher-order interactions. Here, we demonstrate that higher-order Ising machines can solve satisfiability problems more resource-efficiently in terms of the number of spin variables and their connections when compared to traditional second-order Ising machines. Further, our results show on a benchmark dataset of Boolean k-satisfiability problems that higher-order Ising machines implemented with coupled oscillators rapidly find solutions that are better than second-order Ising machines, thus, improving the current state-of-the-art for Ising machines. 

  • 3.
    Dhole, Kaustubh
    et al.
    Emory University, USA; Amelia R&D, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Zhang, Yue
    Westlake Institute for Advanced Study, USA.
    NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation2023Ingår i: NEJLT Northern European Journal of Language Technology, ISSN 2000-1533, Vol. 9, nr 1, s. 1-41Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training datafor natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based naturallanguage (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters(data splits according to specific features). We describe the framework and an initial set of117transformations and23filters for avariety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental humanmistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguousto humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popularlanguage models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases.The infrastructure, datacards, and robustness evaluation results are publicly available onGitHubfor the benefit of researchersworking on paraphrase generation, robustness analysis, and low-resource NLP.

  • 4. Diao, C.
    et al.
    Kleyko, Denis
    Uc Berkeley, USA.
    Rabaey, J. M.
    Olshausen, B. A.
    Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing2021Ingår i: Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc. , 2021, Vol. 2021-JulyKonferensbidrag (Refereegranskat)
    Abstract [en]

    Machine learning algorithms deployed on edge devices must meet certain resource constraints and efficiency requirements. Random Vector Functional Link (RVFL) networks are favored for such applications due to their simple design and training efficiency. We propose a modified RVFL network that avoids computationally expensive matrix operations during training, thus expanding the network’s range of potential applications. Our modification replaces the least-squares classifier with the Generalized Learning Vector Quantization (GLVQ) classifier, which only employs simple vector and distance calculations. The GLVQ classifier can also be considered an improvement upon certain classification algorithms popularly used in the area of Hyperdimensional Computing. The proposed approach achieved state-of-the-art accuracy on a collection of datasets from the UCI Machine Learning Repository-higher than previously proposed RVFL networks. We further demonstrate that our approach still achieves high accuracy while severely limited in training iterations (using on average only 21% of the least-squares classifier computational costs). 

  • 5.
    Frady, E. P.
    et al.
    Intel Labs, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Kymn, C. J.
    University of California, USA.
    Olshausen, B. A.
    University of California, USA.
    Sommer, F. T.
    Intel Labs, USA; University of California, USA.
    Computing on Functions Using Randomized Vector Representations (in brief)2022Ingår i: ACM International Conference Proceeding Series, Association for Computing Machinery , 2022, s. 115-122Konferensbidrag (Refereegranskat)
    Abstract [en]

    Vector space models for symbolic processing that encode symbols by random vectors have been proposed in cognitive science and connectionist communities under the names Vector Symbolic Architecture (VSA), and, synonymously, Hyperdimensional (HD) computing [22, 31, 46]. In this paper, we generalize VSAs to function spaces by mapping continuous-valued data into a vector space such that the inner product between the representations of any two data points approximately represents a similarity kernel. By analogy to VSA, we call this new function encoding and computing framework Vector Function Architecture (VFA). In VFAs, vectors can represent individual data points as well as elements of a function space (a reproducing kernel Hilbert space). The algebraic vector operations, inherited from VSA, correspond to well-defined operations in function space. Furthermore, we study a previously proposed method for encoding continuous data, fractional power encoding (FPE), which uses exponentiation of a random base vector to produce randomized representations of data points and fulfills the kernel properties for inducing a VFA. We show that the distribution from which components of the base vector are sampled determines the shape of the FPE kernel, which in turn induces a VFA for computing with band-limited functions. In particular, VFAs provide an algebraic framework for implementing large-scale kernel machines with random features, extending [51]. Finally, we demonstrate several applications of VFA models to problems in image recognition, density estimation and nonlinear regression. Our analyses and results suggest that VFAs constitute a powerful new framework for representing and manipulating functions in distributed neural systems, with myriad potential applications in artificial intelligence.

  • 6.
    Frady, Edward Paxon
    et al.
    Intel Labs, USA; University of California, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. Intel Labs, USA.
    Sommer, Friedrich T
    Intel Labs, USA; University of California.
    Variable Binding for Sparse Distributed Representations: Theory and Applications2023Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 34, nr 5, s. 2191-2204Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the context of neuroscience and neural networks. 

  • 7.
    Heddes, Mike
    et al.
    University of California, USA.
    Nunes, Igor
    University of California, USA.
    Verges, Pere
    University of California, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap.
    Abraham, Danny
    University of California, USA.
    Givargis, Tony
    University of California, USA.
    Nicolau, Alexandru
    University of California, USA.
    Veidenbaum, Alexander
    University of California, USA.
    Torchhd: An Open Source Python Library to SupportResearch on Hyperdimensional Computing andVector Symbolic Architectures2023Ingår i: Journal of Machine Learning Research, Vol. 24, s. 1-10Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Hyperdimensional computing (HD), also known as vector symbolic architectures (VSA), is a framework for computing with distributed representations by exploiting properties of random high-dimensional vector spaces. The commitment of the scientific community to aggregate and disseminate research in this particularly multidisciplinary area has been fundamental for its advancement. Joining these efforts, we present Torchhd, a highperformance open source Python library for HD/VSA. Torchhd seeks to make HD/VSA more accessible and serves as an efficient foundation for further research and application development. The easy-to-use library builds on top of PyTorch and features state-of-the art HD/VSA functionality, clear documentation, and implementation examples from wellknown publications. Comparing publicly available code with their corresponding Torchhd implementation shows that experiments can run up to 100× faster. Torchhd is available at: https://github.com/hyperdimensional-computing/torchhd.

  • 8.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. UC Berkeley, USA.
    Bybee, C.
    UC Berkeley, USA.
    Kymn, C. J.
    UC Berkeley, USA.
    Olshausen, B. A.
    UC Berkeley, USA.
    Khosrowshahi, A.
    Intel, USA.
    Nikonov, D. E.
    Intel, USA.
    Sommer, F. T.
    Intel, USA.
    Frady, E. P.
    Intel, USA.
    Integer Factorization with Compositional Distributed Representations2022Ingår i: ACM International Conference Proceeding Series, Association for Computing Machinery , 2022, s. 73-80Konferensbidrag (Refereegranskat)
    Abstract [en]

    In this paper, we present an approach to integer factorization using distributed representations formed with Vector Symbolic Architectures. The approach formulates integer factorization in a manner such that it can be solved using neural networks and potentially implemented on parallel neuromorphic hardware. We introduce a method for encoding numbers in distributed vector spaces and explain how the resonator network can solve the integer factorization problem. We evaluate the approach on factorization of semiprimes by measuring the factorization accuracy versus the scale of the problem. We also demonstrate how the proposed approach generalizes beyond the factorization of semiprimes; in principle, it can be used for factorization of any composite number. This work demonstrates how a well-known combinatorial search problem may be formulated and solved within the framework of Vector Symbolic Architectures, and it opens the door to solving similarly difficult problems in other domains.

  • 9.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Bybee, Connor
    University of California, USA.
    Huang, P-C
    University of California, USA.
    Kymn, Christopher
    University of California, USA.
    Olshausen, Bruno
    University of California, USA.
    Frady, E Paxon
    Intel Labs, USA.
    Sommer, Friedrich
    University of California, USA; Intel Labs, USA.
    Efficient Decoding of Compositional Structure in Holistic Representations2023Ingår i: Neural Computation, ISSN 0899-7667, E-ISSN 1530-888X, Vol. 35, nr 7, s. 1159-1186Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks. 

  • 10.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Davies, Mike
    Intel Labs, USA.
    Frady, Paxon
    Intel Labs, USA.
    Kanerva, Pentti
    University of California at Berkeley, USA.
    Kent, Spencer
    University of California at Berkeley, USA.
    Olshausen, Bruno
    University of California at Berkeley, USA.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Rabaey, Jan
    University of California at Berkeley, USA.
    Rachkovskij, Dmitri
    International Research and Training Center for Information Technologies and Systems, Ukraine; Luleå University of Technology, Sweden.
    Rahimi, Abbas
    IBM Research-Zurich, Switzerland.
    Sommer, Friedrich
    University of California at Berkeley, USA.
    Vector Symbolic Architectures as a Computing Framework for Emerging Hardware2022Ingår i: Proceedings of the IEEE, ISSN 0018-9219, E-ISSN 1558-2256, Vol. 110, nr 10, s. 1538-1571Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    This article reviews recent progress in the development of the computing framework vector symbolic architectures (VSA) (also known as hyperdimensional computing). This framework is well suited for implementation in stochastic, emerging hardware, and it naturally expresses the types of cognitive operations required for artificial intelligence (AI). We demonstrate in this article that the field-like algebraic structure of VSA offers simple but powerful operations on high-dimensional vectors that can support all data structures and manipulations relevant to modern computing. In addition, we illustrate the distinguishing feature of VSA, 'computing in superposition,' which sets it apart from conventional computing. It also opens the door to efficient solutions to the difficult combinatorial search problems inherent in AI applications. We sketch ways of demonstrating that VSA are computationally universal. We see them acting as a framework for computing with distributed representations that can play a role of an abstraction layer for emerging computing hardware. This article serves as a reference for computer architects by illustrating the philosophy behind VSA, techniques of distributed computing with them, and their relevance to emerging computing hardware, such as neuromorphic computing. 

  • 11.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Frady, Edward
    Intel Labs, USA; University of California at Berkeley, USA.
    Kheffache, Mansour
    Netlight Consulting AB, Sweden.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware2022Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 33, nr 4, s. 1688-1701Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    We propose an approximation of echo state networks (ESNs) that can be efficiently implemented on digital hardware based on the mathematics of hyperdimensional computing. The reservoir of the proposed integer ESN (intESN) is a vector containing only n-bits integers (where n< 8 is normally sufficient for a satisfactory performance). The recurrent matrix multiplication is replaced with an efficient cyclic shift operation. The proposed intESN approach is verified with typical tasks in reservoir computing: memorizing of a sequence of inputs, classifying time series, and learning dynamic processes. Such architecture results in dramatic improvements in memory footprint and computational efficiency, with minimal performance loss. The experiments on a field-programmable gate array confirm that the proposed intESN approach is much more energy efficient than the conventional ESN. 

  • 12.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Frady, Edward
    University of California at Berkeley, USA; Intel Labs, USA.
    Sommer, Friederich
    University of California at Berkeley, USA; Intel Labs, USA.
    Cellular Automata Can Reduce Memory Requirements of Collective-State Computing2022Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 33, nr 6, s. 2701-2713Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Various nonclassical approaches of distributed information processing, such as neural networks, reservoir computing (RC), vector symbolic architectures (VSAs), and others, employ the principle of collective-state computing. In this type of computing, the variables relevant in computation are superimposed into a single high-dimensional state vector, the collective state. The variable encoding uses a fixed set of random patterns, which has to be stored and kept available during the computation. In this article, we show that an elementary cellular automaton with rule 90 (CA90) enables the space-time tradeoff for collective-state computing models that use random dense binary representations, i.e., memory requirements can be traded off with computation running CA90. We investigate the randomization behavior of CA90, in particular, the relation between the length of the randomization period and the size of the grid, and how CA90 preserves similarity in the presence of the initialization noise. Based on these analyses, we discuss how to optimize a collective-state computing model, in which CA90 expands representations on the fly from short seed patterns--rather than storing the full set of random patterns. The CA90 expansion is applied and tested in concrete scenarios using RC and VSAs. Our experimental results show that collective-state computing with CA90 expansion performs similarly compared to traditional collective-state models, in which random patterns are generated initially by a pseudorandom number generator and then stored in a large memory. 

  • 13.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Karunaratne, Geethan
    IBM Research, Switzerland.
    Rabaey, Jan M.
    IBM Research, Switzerland.
    Sebastian, Abu
    IBM Research, Switzerland.
    Rahimi, Abbas
    University of California, USA.
    Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks2023Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 34, nr 12, s. 10993-10998Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Memory-augmented neural networks enhance a neural network with an external key-value (KV) memory whose complexity is typically dominated by the number of support vectors in the key memory. We propose a generalized KV memory that decouples its dimension from the number of support vectors by introducing a free parameter that can arbitrarily add or remove redundancy to the key memory representation. In effect, it provides an additional degree of freedom to flexibly control the tradeoff between robustness and the resources required to store and compute the generalized KV memory. This is particularly useful for realizing the key memory on in-memory computing hardware where it exploits nonideal, but extremely efficient nonvolatile memory devices for dense storage and computation. Experimental results show that adapting this parameter on demand effectively mitigates up to 44% nonidealities, at equal accuracy and number of devices, without any need for neural network retraining.

  • 14.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Kheffache, Mansour
    Netlight Consulting AB, Sweden.
    Frady, E Paxon
    University of California at Berkeley, USA.
    Wiklund, Urban
    Umeå University, Sweden.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks2021Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 32, nr 8, s. 3777-3783, artikel-id 9174774Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this brief, we focus on resource-efficient randomly connected neural networks known as random vector functional link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world data sets from the UCI machine learning repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small ${n}$ -bits integers, which results in a computationally efficient architecture. Finally, through hardware field-programmable gate array (FPGA) implementations, we show that such an approach consumes approximately 11 times less energy than that of the conventional RVFL.

  • 15.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA .
    Rachkovskij, Dmitri A.
    International Research and Training Center for Information Technologies and Systems, Ukraine; Luleå University of Technology, Sweden.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Rahimi, Abbas
    IBM Research, Switzerland .
    A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations2022Ingår i: ACM Computing Surveys, Vol. 55, nr 6Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    This two-part comprehensive survey is devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and distributed vector representations. Notable models in the HDC/VSA family are Tensor Product Representations, Holographic Reduced Representations, Multiply-Add-Permute, Binary Spatter Codes, and Sparse Binary Distributed Representations but there are other models too. HDC/VSA is a highly interdisciplinary field with connections to computer science, electrical engineering, artificial intelligence, mathematics, and cognitive science. This fact makes it challenging to create a thorough overview of the field. However, due to a surge of new researchers joining the field in recent years, the necessity for a comprehensive survey of the field has become extremely important. Therefore, amongst other aspects of the field, this Part I surveys important aspects such as: known computational models of HDC/VSA and transformations of various input data types to high-dimensional distributed representations. Part II of this survey [ 84 ] is devoted to applications, cognitive computing and architectures, as well as directions for future work. The survey is written to be useful for both newcomers and practitioners.

  • 16.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA; .
    Rachkovskij, Dmitri
    International Research and Training Center for Information Technologies, Ukraine.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Rahimi, Abbas
    Ibm Research Zurich, Switzerland.
    A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges2023Ingår i: ACM Computing Surveys, ISSN 0360-0300, E-ISSN 1557-7341, Vol. 55, nr 9, artikel-id 3558000Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    This is Part II of the two-part comprehensive survey devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Holographic Reduced Representations [321, 326] is an influential HDC/VSA model that is well known in the machine learning domain and often used to refer to the whole family. However, for the sake of consistency, we use HDC/VSA to refer to the field.Part I of this survey [222] covered foundational aspects of the field, such as the historical context leading to the development of HDC/VSA, key elements of any HDC/VSA model, known HDC/VSA models, and the transformation of input data of various types into high-dimensional vectors suitable for HDC/VSA. This second part surveys existing applications, the role of HDC/VSA in cognitive computing and architectures, as well as directions for future work. Most of the applications lie within the Machine Learning/Artificial Intelligence domain; however, we also cover other applications to provide a complete picture. The survey is written to be useful for both newcomers and practitioners. 

    Ladda ner fulltext (pdf)
    fulltext
  • 17.
    Kleyko, Denis
    et al.
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Rosato, Antonello
    University of Rome “La Sapienza”, Italy.
    Frady, Edward Paxon
    Intel Labs, USA.
    Panella, Massimo
    University of Rome “La Sapienza”, Italy.
    Sommer, Friedrich T.
    Intel Labs, USA; University of California at Berkeley, USA.
    Perceptron Theory Can Predict the Accuracy of Neural Networks2023Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. Furthermore, the theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. Moreover, the perceptron theory compares favorably to other methods that do not rely on training an estimator model.

  • 18.
    Osipov, Evgeny
    et al.
    Luleå University of Technology, Sweden.
    Kahawala, S.
    La Trobe University, Australia.
    Haputhanthri, D.
    La Trobe University, Australia.
    Kempitiya, T.
    La Trobe University, Australia.
    Silva, D. D.
    La Trobe University, Australia.
    Alahakoon, D.
    La Trobe University, Australia.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Hyperseed: Unsupervised Learning With Vector Symbolic Architectures2023Ingår i: IEEE Transactions on Neural Networks and Learning Systems, ISSN 2162-237X, E-ISSN 2162-2388, Vol. 12, nr 12, artikel-id e202300141Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Motivated by recent innovations in biologically inspired neuromorphic hardware, this article presents a novel unsupervised machine learning algorithm named Hyperseed that draws on the principles of vector symbolic architectures (VSAs) for fast learning of a topology preserving feature map of unlabeled data. It relies on two major operations of VSA, binding and bundling. The algorithmic part of Hyperseed is expressed within the Fourier holographic reduced representations (FHRR) model, which is specifically suited for implementation on spiking neuromorphic hardware. The two primary contributions of the Hyperseed algorithm are few-shot learning and a learning rule based on single vector operation. These properties are empirically evaluated on synthetic datasets and on illustrative benchmark use cases, IRIS classification, and a language identification task using the $n$ -gram statistics. The results of these experiments confirm the capabilities of Hyperseed and its applications in neuromorphic hardware.

  • 19.
    Rachkovskij, D. A.
    et al.
    Luleå University of Technology, Sweden; International Research and Training Center for Information Technologies and Systems, Ukraine.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California at Berkeley, USA.
    Recursive Binding for Similarity-Preserving Hypervector Representations of Sequences2022Ingår i: 2022 International Joint Conference on Neural Networks (IJCNN), 2022Konferensbidrag (Refereegranskat)
    Abstract [en]

    Hyperdimensional computing (HDC), also known as vector symbolic architectures (VSA), is a computing framework used within artificial intelligence and cognitive computing that operates with distributed vector representations of large fixed dimensionality. A critical step in designing the HDC/VSA solutions is to obtain such representations from the input data. Here, we focus on a wide-spread data type of sequences and propose their transformation to distributed representations that both preserve the similarity of identical sequence elements at nearby positions and are equivariant with respect to the sequence shift. These properties are enabled by forming representations of sequence positions using recursive binding as well as superposition operations. The proposed transformation was experimentally investigated with symbolic strings used for modeling human perception of word similarity. The obtained results are on a par with more sophisticated approaches from the literature. The proposed transformation was designed for the HDC/VSA model known as Fourier Holographic Reduced Representations. However, it can be adapted to some other HDC/VSA models.

  • 20.
    Rosato, Antonello
    et al.
    University of Rome 'La Sapienza', Sweden.
    Panella, Massimo
    University of Rome 'La Sapienza', Sweden.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. Uc Berkeley, USA.
    Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks2021Ingår i: Proceedings of the International Joint Conference on Neural Networks, Vol. 2021-JulyArtikel i tidskrift (Refereegranskat)
    Abstract [en]

    In the supervised learning domain, considering the recent prevalence of algorithms with high computational cost, the attention is steering towards simpler, lighter, and less computationally extensive training and inference approaches. In particular, randomized algorithms are currently having a resurgence, given their generalized elementary approach. By using randomized neural networks, we study distributed classification, which can be employed in situations were data cannot be stored at a central location nor shared. We propose a more efficient solution for distributed classification by making use of a lossy compression approach applied when sharing the local classifiers with other agents. This approach originates from the framework of hyperdimensional computing, and is adapted herein. The results of experiments on a collection of datasets demonstrate that the proposed approach has usually higher accuracy than local classifiers and getting close to the benchmark - the centralized classifier. This work can be considered as the first step towards analyzing the variegated horizon of distributed randomized neural networks.

  • 21.
    Rosato, Antonello
    et al.
    University of Rome 'La Sapienza', Italy.
    Panella, Massimo
    University of Rome 'La Sapienza', Italy.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA; .
    Few-shot Federated Learning in Randomized Neural Networks via Hyperdimensional Computing2022Ingår i: Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc. , 2022Konferensbidrag (Refereegranskat)
    Abstract [en]

    The recent interest in federated learning has initiated the investigation for efficient models deployable in scenarios with strict communication and computational constraints. Furthermore, the inherent privacy concerns in decentralized and federated learning call for efficient distribution of information in a network of interconnected agents. Therefore, we propose a novel distributed classification solution that is based on shallow randomized networks equipped with a compression mechanism that is used for sharing the local model in the federated context. We make extensive use of hyperdimensional computing both in the local network model and in the compressed communication protocol, which is enabled by the binding and the superposition operations. Accuracy, precision, and stability of our proposed approach are demonstrated on a collection of datasets with several network topologies and for different data partitioning schemes.

  • 22.
    Rosato, Antonello
    et al.
    University of Rome “La Sapienza”, Italy.
    Panella, Massimo
    University of Rome “La Sapienza”, Italy.
    Osipov, Evgeny
    Luleå University of Technology, Sweden.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks2021Ingår i: Lecture Notes in Computer Science, ISSN 0302-9743, E-ISSN 1611-3349, Vol. 12862 LNCS, s. 155-167Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    A change of the prevalent supervised learning techniques is foreseeable in the near future: from the complex, computational expensive algorithms to more flexible and elementary training ones. The strong revitalization of randomized algorithms can be framed in this prospect steering. We recently proposed a model for distributed classification based on randomized neural networks and hyperdimensional computing, which takes into account cost of information exchange between agents using compression. The use of compression is important as it addresses the issues related to the communication bottleneck, however, the original approach is rigid in the way the compression is used. Therefore, in this work, we propose a more flexible approach to compression and compare it to conventional compression algorithms, dimensionality reduction, and quantization techniques.

  • 23.
    Rutqvist, D.
    et al.
    BnearIT AB, Sweden.
    Kleyko, Denis
    Luleå University of Technology, Sweden.
    Blomstedt, F.
    BnearIT AB, Sweden.
    An automated machine learning approach for smart waste management systems2020Ingår i: IEEE Transactions on Industrial Informatics, ISSN 1551-3203, E-ISSN 1941-0050, Vol. 16, nr 1, s. 384-392, artikel-id 8709695Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    This paper presents the use of automated machine learning for solving a practical problem of a real-life Smart Waste Management system. In particular, the focus of the paper is on the problem of detection (i.e., binary classification) of emptying of a recycling container using sensor measurements. Numerous data-driven methods for solving the problem are investigated in a realistic setting where most of the events are not actual emptying. The investigated methods include the existing manually engineered model and its modification as well as conventional machines learning algorithms. The use of machine learning allows improving the classification accuracy and recall of the existing manually engineered model from 86.8% and 47.9% to 99.1% and 98.2%, respectively, when using the best performing solution. This solution uses a Random Forest classifier on a set of features based on the filling level at different given time spans. Finally, compared to the baseline existing manually engineered model, the best performing solution also improves the quality of forecasts for emptying time of recycling containers. 

  • 24.
    Schlegel, Kenny
    et al.
    Chemnitz University of Technology, Germany.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. Örebro University, Sweden.
    Brinkmann, Benjamin H.
    Mayo Clinic Rochester, USA.
    Nurse, Ewan S.
    Seer Medical, Australia.
    Gayler, Ross W.
    Independent researcher, Australia.
    Neubert, Peer
    University of Koblenz, Germany.
    Lessons from a challenge on forecasting epileptic seizures from non-cerebral signals2024Ingår i: Nature Machine Intelligence, ISSN 2522-5839, Vol. 6, nr 2, s. 243-244Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    The “My Seizure Gauge” competition explored the challenge of forecasting epileptic seizures using non-invasive wearable devices without an electroencephalogram. The organizers and the winning team reflect on their experiences. 

  • 25.
    Srivastava, Aarohi
    et al.
    University of Notre Dame, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap.
    Wu, Ziyi
    Beyond the Imitation Game: Quantifying and extrapolatingthe capabilities of language models2023Ingår i: Transactions on Machine Learning Research, E-ISSN 2835-8856, nr 5Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG- bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood develop- ment, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google- internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

  • 26.
    Teeters, Jeffrey
    et al.
    University of California, USA.
    Kleyko, Denis
    RISE Research Institutes of Sweden, Digitala system, Datavetenskap. University of California, USA.
    Kanerva, Pentti
    University of California, USA.
    Olshausen, Bruno
    University of California, USA.
    On separating long- and short-term memories in hyperdimensional computing2023Ingår i: Frontiers in Neuroscience, ISSN 1662-4548, E-ISSN 1662-453X, Vol. 16, artikel-id 867568Artikel i tidskrift (Refereegranskat)
    Abstract [en]

    Operations on high-dimensional, fixed-width vectors can be used to distribute information from several vectors over a single vector of the same width. For example, a set of key-value pairs can be encoded into a single vector with multiplication and addition of the corresponding key and value vectors: the keys are bound to their values with component-wise multiplication, and the key-value pairs are combined into a single superposition vector with component-wise addition. The superposition vector is, thus, a memory which can then be queried for the value of any of the keys, but the result of the query is approximate. The exact vector is retrieved from a codebook (a.k.a. item memory), which contains vectors defined in the system. To perform these operations, the item memory vectors and the superposition vector must be the same width. Increasing the capacity of the memory requires increasing the width of the superposition and item memory vectors. In this article, we demonstrate that in a regime where many (e.g., 1,000 or more) key-value pairs are stored, an associative memory which maps key vectors to value vectors requires less memory and less computing to obtain the same reliability of storage as a superposition vector. These advantages are obtained because the number of storage locations in an associate memory can be increased without increasing the width of the vectors in the item memory. An associative memory would not replace a superposition vector as a medium of storage, but could augment it, because data recalled from an associative memory could be used in algorithms that use a superposition vector. This would be analogous to how human working memory (which stores about seven items) uses information recalled from long-term memory (which is much larger than the working memory). We demonstrate the advantages of an associative memory experimentally using the storage of large finite-state automata, which could model the storage and recall of state-dependent behavior by brains. 

1 - 26 av 26
RefereraExporteraLänk till träfflistan
Permanent länk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf