An optimized recursive learning algorithm for three-layer feedforward neural networks for mimo nonlinear system identifications

Back-propagation with gradient method is the most popular learning algorithm for feed-forward neural networks. However, it is critical to determine a proper fixed learning rate for the algorithm. In this paper, an optimized recursive algorithm is presented for online learning based on matrix operation and optimization methods analytically, which can avoid the trouble to select a proper learning rate for the gradient method. The proof of weak convergence of the proposed algorithm also is given. Although this approach is proposed for three-layer, feed-forward neural networks, it could be extended to multiple layer feed-forward neural networks. The effectiveness of the proposed algorithms applied to the identification of behavior of a two-input and two-output non-linear dynamic system is demonstrated by simulation experiments.

💡 Research Summary

The paper addresses a fundamental drawback of the classic back‑propagation (BP) algorithm with a fixed learning rate: selecting an appropriate step size is both critical and cumbersome, especially for online learning of nonlinear multiple‑input multiple‑output (MIMO) systems. To eliminate this bottleneck, the authors propose an Optimized Recursive Learning Algorithm (ORLA) that derives an analytical, matrix‑based weight update rule using gradient and (approximate) Hessian information, thereby removing the need for an explicit learning‑rate hyper‑parameter.

Algorithmic Core
Instead of updating each weight scalar‑wise, ORLA treats the entire weight matrix (W) as a single variable. At iteration (k) the gradient (g_k = \nabla J(W_k)) and an approximation of the Hessian (H_k) are computed. The update follows a quasi‑Newton step:

\