Title: Neural Distributed Autoassociative Memories: A Survey
ArXiv ID: 1709.00848
Date: 2017-09-05
Authors: Researchers from original ArXiv paper
📝 Abstract
Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension. The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons). Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints. Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors.
💡 Deep Analysis
Deep Dive into Neural Distributed Autoassociative Memories: A Survey.
Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension. The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons). Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite gra
📄 Full Content
Neural Distributed Autoassociative
Memories: A Survey
V.I. Gritsenko, D.A. Rachkovskij, A.A. Frolov, R. Gayler, D. Kleyko, E. Osipov
Abstract
Introduction. Neural network models of autoassociative, distributed memory allow storage
and retrieval of many items (vectors) where the number of stored items can exceed the vector
dimension (the number of neurons in the network). This opens the possibility of a sublinear
time search (in the number of stored items) for approximate nearest neighbors among vectors
of high dimension.
The purpose of this paper is to review models of autoassociative, distributed memory
that can be naturally implemented by neural networks (mainly with local learning rules and
iterative dynamics based on information locally available to neurons).
Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts,
that have connections between pairs of neurons and operate on sparse binary vectors. We dis-
cuss not only autoassociative memory, but also the generalization properties of these net-
works. We also consider neural networks with higher-order connections and networks with a
bipartite graph structure for non-binary data with linear constraints.
Conclusions. In conclusion we discuss the relations to similarity search, advantages and
drawbacks of these techniques, and topics for further research. An interesting and still not
completely resolved question is whether neural autoassociative memories can search for ap-
proximate nearest neighbors faster than other index structures for similarity search, in partic-
ular for the case of very high dimensional vectors.
Keywords: distributed associative memory, sparse binary vector, Hopfield network,
Willshaw memory, Potts model, nearest neighbor, similarity search.
DOI: https://doi.org/10.15407/kvt188.02.005
This front page has been added to assist automated indexing.
The paper as published begins on the next page.
Please cite as:
V.I. Gritsenko, D.A. Rachkovskij, A.A. Frolov, R. Gayler, D. Kleyko, E. Osipov.
Neural distributed autoassociative memories: A survey. Cybernetics and Computer
Engineering. 2017. N 2 (188). P. 5–35.
ISSN 2519-2205 (Online), ISSN 0454-9910 (Print). Киб. и выч. техн. 2017. № 2 (188)
Информатика
и информационные технологии
DOI: https://doi.org/10.15407/kvt188.02.005
УДК 004.22 + 004.93'11
V.I. GRITSENKO1, Corresponding Member of NAS of Ukraine, Director,
e-mail: vig@irtc.org.ua
D.A. RACHKOVSKIJ1, Dr (Engineering), Leading Researcher, Dept. of Neural Information Processing Technologies,
e-mail: dar@infrm.kiev.ua A.A. FROLOV2, Dr (Biology), Professor, Faculty of Electrical Engineering and Computer Science FEI,
e-mail: docfact@gmail.com R. GAYLER3, PhD (Psychology), Independent Researcher, e-mail: r.gayler@gmail.com
D. KLEYKO4 graduate student, Department of Computer Science, Electrical and Space Engineering,
e-mail: denis.kleyko@ltu.se
E. OSIPOV4 PhD (Informatics), Professor,
Department of Computer Science, Electrical and Space Engineering,
e-mail: evgeny.osipov@ltu.se 1 International Research and Training Center for Information Technologies
and Systems of the NAS of Ukraine and of Ministry of Education
and Science of Ukraine, ave. Acad. Glushkova, 40, Kiev, 03680, Ukraine
2 Technical University of Ostrava, 17 listopadu 15, 708 33 Ostrava-Poruba,
Czech Republic
3 Melbourne, VIC, Australia
4 Lulea University of Technology, 971 87 Lulea, Sweden
NEURAL DISTRIBUTED AUTOASSOCIATIVE MEMORIES: A SURVEY
Introduction. Neural network models of autoassociative, distributed memory allow storage
and retrieval of many items (vectors) where the number of stored items can exceed the vector
dimension (the number of neurons in the network). This opens the possibility of a sublinear
time search (in the number of stored items) for approximate nearest neighbors among vectors
of high dimension.
The purpose of this paper is to review models of autoassociative, distributed memory
that can be naturally implemented by neural networks (mainly with local learning rules and
iterative dynamics based on information locally available to neurons).
Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts,
that have connections between pairs of neurons and operate on sparse binary vectors. We
5
V.I. GRITSENKO, D.A. RACHKOVSKIJ, A.A. FROLOV, R. GAYLER, D. KLEYKO, E. OSIPOV, 2017
V.I. Gritsenko, D.A. Rachkovskij, A.A. Frolov, R. Gayler, D. Kleyko, E. Osipov
ISSN 2519-2205 (Online), ISSN 0454-9910 (Print). Киб. и выч. техн. 2017. № 2 (188)
6
discuss not only autoassociative memory, but also the generalization properties of these
networks. We also consider neural networks with higher-order connections and networks
with a bipartite graph structure for non-binary data with linear constraints.
Conclusions. In conclusion we discuss the relations to similarity search, advantages
and drawbacks of these techniques, and topics for further rese
