HyperEmbed: Tradeoffs between Resources and Performance in NLP Tasks with Hyperdimensional Computing Enabled Embedding of n-gram StatisticsShow others and affiliations
2021 (English)In: Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc. , 2021, Vol. 2021-JulyConference paper, Published paper (Refereed)
Abstract [en]
Recent advances in Deep Learning have led to a significant performance increase on several NLP tasks, however, the models become more and more computationally demanding. Therefore, this paper tackles the domain of computationally efficient algorithms for NLP tasks. In particular, it investigates distributed representations of n -gram statistics of texts. The representations are formed using hyperdimensional computing enabled embedding. These representations then serve as features, which are used as input to standard classifiers. We investigate the applicability of the embedding on one large and three small standard datasets for classification tasks using nine classifiers. The embedding achieved on par F_1 scores while decreasing the time and memory requirements by several times compared to the conventional n -gram statistics, e.g., for one of the classifiers on a small dataset, the memory reduction was 6.18 times; while train and test speed-ups were 4.62 and 3.84 times, respectively. For many classifiers on the large dataset, memory reduction was ca. 100 times and train and test speed-ups were over 100 times. Importantly, the usage of distributed representations formed via hyperdimensional computing allows dissecting strict dependency between the dimensionality of the representation and n-gram size, thus, opening a room for tradeoffs.
Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc. , 2021. Vol. 2021-July
Keywords [en]
Commerce; Deep learning; Embeddings; Large dataset; Natural language processing systems; Statistical tests, Distributed representation; Embeddings; Gram statistic; Hyperdimensional computing; Intent classification; Memory reduction; N-gram statistics; Performance; Test speed; Train’s speed, Classification (of information)
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:ri:diva-68292DOI: 10.1109/IJCNN52387.2021.9534359Scopus ID: 2-s2.0-85108654382OAI: oai:DiVA.org:ri-68292DiVA, id: diva2:1817591
Conference
International Joint Conference on Neural Networks, IJCNN 2021 Virtual, Shenzhen, China. 18 July 2021 through 22 July 2021
2023-12-062023-12-062023-12-12Bibliographically approved