Networks are fundamental building blocks for representing data, and computations. Remarkable progress in learning in structurally defined (shallow or deep) networks has recently been achieved. Here we introduce evolutionary exploratory search and learning method of topologically flexible networks under the constraint of producing elementary computational steady-state input-output operations. Our results include; (1) the identification of networks, over four orders of magnitude, implementing computation of steady-state input-output functions, such as a band-pass filter, a threshold function, and an inverse band-pass function. Next, (2) the learned networks are technically controllable as only a small number of driver nodes are required to move the system to a new state. Furthermore, we find that the fraction of required driver nodes is constant during evolutionary learning, suggesting a stable system design. (3), our framework allows multiplexing of different computations using the same network. For example, using a binary representation of the inputs, the network can readily compute three different input-output functions. Finally, (4) the proposed evolutionary learning demonstrates transfer learning. If the system learns one function A, then learning B requires on average less number of steps as compared to learning B from tabula rasa. We conclude that the constrained evolutionary learning produces large robust controllable circuits, capable of multiplexing and transfer learning. Our study suggests that network-based computations of steady-state functions, representing either cellular modules of cell-to-cell communication networks or internal molecular circuits communicating within a cell, could be a powerful model for biologically inspired computing. This complements conceptualizations such as attractor based models, or reservoir computing.
A Turing machine (TM) is a universal mathematical model of computation. Given a table of rules and a tape, a TM is essentially a discrete system defining a function to be computable if and only if it can be computed on a TM [1]. In practice, a concept of learning is executed as an algorithm which in turn is realized in an architecture. For example, inspired by neural circuits, realized as (deep) neural feed-forward network models of computation [2], learning acts on the connections. Attractor dynamics is another popular conception of neural [3] computations where the fixed points correspond to memory states or to cell-types as in the case of molecular (gene) based computations [4].
Here we are inspired by the computational processing power in cells in living systems, where networks of molecular components collectively transform input signals to the cell into outputs at the cellular level. In the case of neurons, we can represent such a computation as a threshold computation using a ReLU or softmax unit. Yet, we do not understand the underlying network of molecular computations occurring within different cells, realizing different input-output transformations. In this paper, we ask if using an evolutionary search to identify networks capable of computing different input-output functions. Specifically, we address whether such a system can represent different input-output computations in the same network, whether the computations are controllable in an engineering sense, and if transfer learning can occur under such constraints. We formulate a meta-functional networks learning (mFNL) scheme with adopted gradient-based optimization [5], a Simple Genetic Algorithm (SGA) [6] and three milestone Evolutionary Algorithms (EAs): Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [7], Standard Particle Swarm Optimization (SPSO2011) [8], and Ant Colony Optimization for Continuous Domains (ACOR) [9]. Hence, we search for new architectures for computing under the soft constraint that the networks should incorporate a couple of fundamental primitives defined as specified input-output functions. While machine learning techniques such as transductive and inductive learning dealing with graphs have recently become mainstream research focus on artificial intelligence and machine learning [10,11,12], we have still limited understanding of the space of putative network architectures for computing. The generative large network models, identified by the evolutionary search, hold the promise to be of broad interest since networks are used as fundamental building blocks for representing data, and computations in biology, social sciences, and communication networks [13]. Our work is predicated on the notion that by imposing fundamental computational operations (input-output transformations), we can generate underlying ensembles of networks realizing such computations.
Node Dynamics: To compute input-output functions from a network, we equip the intrinsic network structure with a set of dynamical equations, defining how a given node interacts with its neighbors. The form of such equations is qualitatively motivated by chemical reactions and molecular regulatory control systems such as genes in a cell. A non-linear threshold function summing input from nearby nodes defines the node dynamics as:
For each node i, y i represents the activation or expression rate, f is a nonlinear sigmoid function: f (x) = σ(x) = 1 1+e -x , I i is the input matrix prescribed into the system, and both k 1i and k 2i are constant vectors which represent the maximum expression and kinetic degradation rate respectively. Note that the W ij represents the influence from nodes j to nodes i. For simplicity, we have set all elements of k 1i and k 2i to 1 for all of our experiments. The motivation originates from the Hopfield network [3], where we follow the modifications as described in previous works [14,15,16,5]. We solve the ODE numerically using the forward Euler’s method in which y n+1 = y n + g(y n , t n )δt and δt is set to 1 (we have checked that the results do not change for smaller values of δt). A network of such connected units collectively computes a given steady-state input-output function. Equation 1 represents a special version of a recurrent neural network [17] and the fully connected network structure looks deceptively similar to a Boltzmann Machine [18] in which the two visible nodes are the input and output nodes respectively while all other hidden nodes (equipped with the above dynamics) are the supporting nodes. However, the links are directed and mutually influencing between nodes for learning appropriate links in order to implement the desired input-output function from the system, see Figure 1a.
Loss function for computational input-output functions: We use an “L2-norm [19] error function between the output level points of the simulated function and the desired function”. Then, if the simulated function is close to the desired fun
This content is AI-processed based on open access ArXiv data.
