Efficient supervised learning in networks with binary synapses

Recent experimental studies indicate that synaptic changes induced by neuronal activity are discrete jumps between a small number of stable states. Learning in systems with discrete synapses is known to be a computationally hard problem. Here, we study a neurobiologically plausible on-line learning algorithm that derives from Belief Propagation algorithms. We show that it performs remarkably well in a model neuron with binary synapses, and a finite number of hidden' states per synapse, that has to learn a random classification task. Such system is able to learn a number of associations close to the theoretical limit, in time which is sublinear in system size. This is to our knowledge the first on-line algorithm that is able to achieve efficiently a finite number of patterns learned per binary synapse. Furthermore, we show that performance is optimal for a finite number of hidden states which becomes very small for sparse coding. The algorithm is similar to the standard perceptron’ learning algorithm, with an additional rule for synaptic transitions which occur only if a currently presented pattern is `barely correct’. In this case, the synaptic changes are meta-plastic only (change in hidden states and not in actual synaptic state), stabilizing the synapse in its current state. Finally, we show that a system with two visible states and K hidden states is much more robust to noise than a system with K visible states. We suggest this rule is sufficiently simple to be easily implemented by neurobiological systems or in hardware.

💡 Research Summary

The paper addresses the long‑standing challenge of learning with binary synapses, a problem that is computationally hard yet biologically relevant because experimental evidence suggests that real synapses switch between a small number of discrete, stable states. The authors propose a biologically plausible online learning algorithm inspired by belief‑propagation (BP) methods, called the BPI (BP‑Inspired) algorithm, and a stochastic variant SBPI. Each synapse possesses a binary weight wᵢ = sign(hᵢ) but also a hidden integer variable hᵢ that can take multiple values. The hidden variable determines the sign of the weight while providing a set of internal “meta‑plastic” states that can be adjusted without immediately changing the observable synaptic efficacy.

Learning proceeds by presenting random input patterns ξᵢ (±1 or 0/1) and computing the total input I = Σᵢ wᵢ ξᵢ. Three update rules are applied:
(R1) If I > 1, the output is safely correct and no changes are made.
(R2) If I = 1 (the output is “barely correct”), only those synapses that contributed positively (hᵢ·ξᵢ ≥ 1) have their hidden variable incremented by 2·ξᵢ; this is a meta‑plastic adjustment that makes the synapse less likely to flip in the future.
(R3) If I ≤ –1 (the output is wrong), all synapses are updated by adding 2·ξᵢ to their hidden variables, which may flip the sign of wᵢ.

The deterministic BPI algorithm (R2 always applied) achieves modest capacity (α ≈ 0.3, where α = p/N, p = number of patterns) but converges extremely fast, with the number of presentations per pattern scaling roughly as (log N)¹·⁵. The stochastic SBPI algorithm applies rule R2 with probability pₛ; by tuning pₛ ≈ 0.3 the system reaches a much higher capacity (α ≈ 0.65), close to the theoretical limit for binary perceptrons (≈0.83). Importantly, convergence time remains sub‑linear in N, often logarithmic, which is dramatically better than exhaustive search (exponential) or traditional perceptron learning with clipped weights (which becomes exponential in N).

The authors also explore the effect of limiting the hidden variable to a finite set of K states. In sparse coding regimes (low activity fraction), optimal performance is achieved with surprisingly small K (as low as 3–5). Moreover, a network with two visible states (±1) and K hidden states per synapse is far more robust to random synaptic noise than a network that directly uses K visible weight levels. This robustness stems from the meta‑plastic mechanism that stabilizes the current binary state while allowing hidden variables to absorb fluctuations.

Computationally, each pattern update requires O(N) operations, and the total learning time grows sub‑linearly with network size, making the algorithm suitable for large‑scale implementations. The simplicity of the update rules (only local information: presynaptic activity, current post‑synaptic output, and an error signal) suggests that the algorithm could be realized in biological circuits or neuromorphic hardware.

In summary, the paper demonstrates that binary synapses equipped with a modest number of hidden states, together with a belief‑propagation‑inspired online rule that includes a “barely correct” meta‑plastic adjustment, can learn near‑optimal numbers of random associations efficiently. This bridges the gap between theoretical capacity results for binary perceptrons and practical, biologically plausible learning mechanisms, opening avenues for both neuroscience (understanding synaptic meta‑plasticity) and engineering (design of low‑precision, high‑capacity learning devices).