Hybrid Neural Network Architecture for On-Line Learning

Yuhua Chen*† · Subhash Kak‡ · Lei Wang§

Abstract Approaches to machine intelligence based on brain models have stressed the use of neural networks for generalization. Here we propose the use of a hybrid neural network architecture that uses two kind of neural networks simultaneously: (i) a surface learning agent that quickly adapt to new modes of operation; and, (ii) a deep learning agent that is very accurate within a specific regime of operation. The two networks of the hybrid architecture perform complementary functions that improve the overall performance. The performance of the hybrid architecture has been compared with that of back-propagation perceptrons and the CC and FC networks for chaotic time-series prediction, the CATS benchmark test, and smooth function approximation. It has been shown that the hybrid architecture provides a superior performance based on the RMS error criterion.

Keywords Neural Networks · Instantaneously trained networks · Back- Propagation · On-line learning

1 Introduction On-line learning requires the learning of the mode or context, out of a set of many, within which the time-varying system is evolving. A few examples of time- varying systems are: aircraft during flight since the configurations of control surfaces as well as flight conditions and the weight of the aircraft are continually changing; the human vocal tract, as the shape of the vocal organs gets modified in the production of different vocalizations; electric or computer networks which may have different modes of behavior depending of the time of the day; and the financial system which changes according to the investor sentiment. In this paper, we consider such non-standard pattern recognition and discovery problems where statistical techniques and conventional neural networks are not convenient to use because of their slow speed.
Pattern discovery may be based on statistical methods [1], [2] or on neural networks. The currently researched techniques include various kinds of neural networks [3], principal-component analysis [4], principal-component regression

• Corresponding author, e-mail: yuhua.chen@mail.uh.edu † Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77204-4005, USA ‡ Department of Computer Science, Oklahoma State University, Stillwater, OK 74078, USA § Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77204-4005, USA

2 [4], partial least-squares regression [5], and combinatorial methods [6]. The Vapnik-Chervonenkis dimension [1] quantifies the ease of learning categories from small data sets, and if the dimension is finite, machine learning of a certain kind can be efficiently done. The time-varying nature of the data sets that we wish to consider here precludes this approach. Machine intelligence that is based on brain models has stressed the use of neural networks [7] that learn from a training set and then generalize on new input data. However, it has been argued [8], [9] that such models miss on the short-term learning component of biological systems which is a very important component in adaptation to changing environment. Biological systems operate in time-changing environments, and they are especially good at learning in such conditions. Biological learning appears to be at the basis of the capacity of biological systems to adapt to time-varying environments [10],[11]. The learning components of biological systems may be divided into three types: sensory, short-term and long-term [12]. The sensory component provides immediate simple reactions to certain changes in the environment. One example of the sensory component is human reflex, which is almost instant in response to stimulus. The long-term learning component is based on experiences gained over periods of time. The short-term learning stands in between the sensory and long-term learning components, and is capable of performing more complex learning tasks than the sensory component while being able to adapt to the changing environment quickly.
The interplay among these biological learning components appears to provide to the organism the capacity to operate in uncertain environments. In addition, two cognitive processes that help in preventing overloading are sensory gating (by which the brain adjusts its response to stimuli) and selective attention (which allows the brain to concentrate on one aspect to the exclusion of others). This is seen, for example, in the driving of a car, where the driver may not be paying total attention to the road in ordinary circumstances. But when the driving conditions are difficult, a different center takes over the driving completely. The first of the two cases may be ascribed to long-term learning and the second to short-term learning. In

…(Full text truncated)…