Efficient supervised learning in networks with binary synapses
📝 Abstract
Recent experimental studies indicate that synaptic changes induced by neuronal activity are discrete jumps between a small number of stable states. Learning in systems with discrete synapses is known to be a computationally hard problem. Here, we study a neurobiologically plausible on-line learning algorithm that derives from Belief Propagation algorithms. We show that it performs remarkably well in a model neuron with binary synapses, and a finite number of hidden' states per synapse, that has to learn a random classification task. Such system is able to learn a number of associations close to the theoretical limit, in time which is sublinear in system size. This is to our knowledge the first on-line algorithm that is able to achieve efficiently a finite number of patterns learned per binary synapse. Furthermore, we show that performance is optimal for a finite number of hidden states which becomes very small for sparse coding. The algorithm is similar to the standard perceptron’ learning algorithm, with an additional rule for synaptic transitions which occur only if a currently presented pattern is `barely correct’. In this case, the synaptic changes are meta-plastic only (change in hidden states and not in actual synaptic state), stabilizing the synapse in its current state. Finally, we show that a system with two visible states and K hidden states is much more robust to noise than a system with K visible states. We suggest this rule is sufficiently simple to be easily implemented by neurobiological systems or in hardware.
💡 Analysis
Recent experimental studies indicate that synaptic changes induced by neuronal activity are discrete jumps between a small number of stable states. Learning in systems with discrete synapses is known to be a computationally hard problem. Here, we study a neurobiologically plausible on-line learning algorithm that derives from Belief Propagation algorithms. We show that it performs remarkably well in a model neuron with binary synapses, and a finite number of hidden' states per synapse, that has to learn a random classification task. Such system is able to learn a number of associations close to the theoretical limit, in time which is sublinear in system size. This is to our knowledge the first on-line algorithm that is able to achieve efficiently a finite number of patterns learned per binary synapse. Furthermore, we show that performance is optimal for a finite number of hidden states which becomes very small for sparse coding. The algorithm is similar to the standard perceptron’ learning algorithm, with an additional rule for synaptic transitions which occur only if a currently presented pattern is `barely correct’. In this case, the synaptic changes are meta-plastic only (change in hidden states and not in actual synaptic state), stabilizing the synapse in its current state. Finally, we show that a system with two visible states and K hidden states is much more robust to noise than a system with K visible states. We suggest this rule is sufficiently simple to be easily implemented by neurobiological systems or in hardware.
📄 Content
arXiv:0707.1295v1 [q-bio.NC] 9 Jul 2007 Efficient supervised learning in networks with binary synapses Carlo Baldassi ISI Foundation, Viale S. Severo 65, I-10133 Torino, Italy Alfredo Braunstein Politecnico di Torino, C.so Duca degli Abruzzi 24, I-10129 Torino, Italy and ISI Foundation, Viale S. Severo 65, I-10133 Torino, Italy Nicolas Brunel Laboratory of Neurophysics and Physiology (UMR 8119), CNRS-Universit´e Paris 5 Ren´e Descartes, 45 rue des Saints P`eres, 75270 Paris Cedex 06 and ISI Foundation, Viale S. Severo 65, I-10133 Torino, Italy Riccardo Zecchina Politecnico di Torino, C.so Duca degli Abruzzi 24, I-10129 Torino, Italy and ICTP, Strada Costiera 11, I-34100 Trieste, Italy Recent experimental studies indicate that synaptic changes induced by neuronal activity are discrete jumps between a small number of stable states. Learning in systems with discrete synapses is known to be a computationally hard problem. Here, we study a neurobiologically plausible on- line learning algorithm that derives from Belief Propagation algorithms. We show that it performs remarkably well in a model neuron with binary synapses, and a finite number of ‘hidden’ states per synapse, that has to learn a random classification task. Such system is able to learn a number of associations close to the theoretical limit, in time which is sublinear in system size. This is to our knowledge the first on-line algorithm that is able to achieve efficiently a finite number of patterns learned per binary synapse. Furthermore, we show that performance is optimal for a finite number of hidden states which becomes very small for sparse coding. The algorithm is similar to the standard ‘perceptron’ learning algorithm, with an additional rule for synaptic transitions which occur only if a currently presented pattern is ‘barely correct’. In this case, the synaptic changes are meta-plastic only (change in hidden states and not in actual synaptic state), stabilizing the synapse in its current state. Finally, we show that a system with two visible states and K hidden states is much more robust to noise than a system with K visible states. We suggest this rule is sufficiently simple to be easily implemented by neurobiological systems or in hardware. I. INTRODUCTION Learning and memory are widely believed to occur through mechanisms of synaptic plasticity. In spite of a huge amount of experimental data documenting various forms of plasticity, as e.g. long-term potentiation (LTP) and long- term depression (LTD), the mechanisms by which a synapse changes its efficacy, and those by which it can maintain these changes over time remain unclear. Recent experiments have suggested single synapses could be similar to noisy binary switches [1, 2]. Bistability could be in principle induced by positive feedback loops in protein interaction networks of the post-synaptic density [3, 4, 5]. Binary synapses would have the advantage of robustness to noise and hence could preserve memory over long time scales, compared to analog systems which are typically much more sensitive to noise. Many neural network models of memory use binary synapses to store information [6, 7, 8, 9, 10, 11, 12]. In some of these network models, learning occurs in an unsupervised way. From the point of view of a single synapse, this means that transitions between the two synaptic states (a state of low or zero efficacy, and a state of high efficacy) are induced by pre and post-synaptic activity alone. Tsodyks [14] and Amit and Fusi [9, 10] have shown that the performance of such systems (in terms of information stored per synapse) is very poor, unless two conditions are met: (1) activity in the network is sparse (very low fraction of neurons active at a given time); and (2) transitions are stochastic, with in average a balance between up (LTP-like) and down (LTD-like) transitions. This poor performance has motivated further studies [12] in which hidden states are added to the synapse in order to provide it with a multiplicity of time scales, allowing for both fast learning and slow forgetting. In a supervised learning scenario, synaptic modifications are induced not only by the activity of pre and post- synaptic neurons but also by an additional ‘teacher’ or ‘error’ signal which gates the synaptic modifications. The prototypical network in which this type of learning has been studied is the one-layer perceptron which has to perform a set of input-output associations, i.e. learn to classify correctly input patterns in two classes. In the case of analog synapses, algorithms are known to converge to synaptic weights that solve the task, provided such weights exist 2 [15, 16]. On the other hand, no efficient algorithms are known to exist in a perceptron with binary (or more generally with a finite number of states) synapses, in the case the number of patterns to be learned scales with the number of synapses. In fact, studies on the capacity of binary perceptrons [18, 19] used complete enumeration schemes in order t
This content is AI-processed based on ArXiv data.