Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.

💡 Research Summary

This paper addresses the challenge of training deep multidimensional recurrent neural networks (MDRNNs) for sequence labeling tasks such as handwriting and speech recognition. While MDRNNs have demonstrated strong performance, their depth has historically been limited to around five layers due to difficulties associated with vanishing/exploding gradients and the non‑convex nature of the Connectionist Temporal Classification (CTC) loss. The authors propose using Hessian‑Free (HF) optimization, a second‑order method that leverages curvature information without explicitly forming the Hessian matrix. HF relies on the Generalized Gauss‑Newton (GGN) approximation, which is positive semi‑definite when the loss function is convex (e.g., softmax with cross‑entropy). However, CTC combines softmax outputs across all possible label paths via a log‑sum‑exp operation, rendering the overall loss non‑convex and unsuitable for direct GGN application.
To overcome this, the paper reformulates the CTC objective by separating it into a non‑convex mapping (N_c) (which aggregates path scores) and a convex component (L_c) (a cross‑entropy over the aggregated scores). By further linearizing (L_c) to obtain (L_p), the authors derive a block‑diagonal approximation of the Hessian where each block corresponds to a single time step and has the familiar form (\text{diag}(y_t) - y_t y_t^\top) with (y_t) being the softmax output at time (t). This block‑diagonal structure enables efficient computation of the matrix‑vector product (Gv) required by the conjugate‑gradient sub‑routine within HF.
The paper also establishes a theoretical connection between the proposed approximation, the Fisher information matrix, and the Expectation‑Maximization (EM) algorithm. Under the condition that a dominant path dominates the probability mass for each label sequence, the Hessian of the approximated loss coincides with the Fisher matrix, guaranteeing positive semi‑definiteness and statistical optimality.
Empirically, the authors train MDRNNs with depths ranging from 1 to 15 layers on offline handwriting and phoneme recognition datasets. Using HF together with the convex CTC approximation, they observe a consistent reduction in error rates as depth increases, with the 15‑layer model achieving more than a 10 % relative improvement over the 5‑layer baseline. Importantly, this performance gain is realized without any layer‑wise pre‑training or bootstrapping techniques, demonstrating that HF can effectively navigate the highly non‑linear loss landscape of deep recurrent architectures.
In summary, the work introduces a practical framework for training very deep MDRNNs by (1) applying Hessian‑Free optimization, (2) constructing a convex surrogate for the CTC loss that yields a tractable GGN/Fisher matrix, and (3) validating the approach on real‑world sequence labeling tasks. The results suggest that deep MDRNNs, previously constrained by optimization difficulties, can now be leveraged for complex multidimensional data such as video, medical imaging, and multi‑sensor time series, opening new avenues for research and application.