Effect of indirect dependencies on "A mutual information minimization approach for a class of nonlinear recurrent separating systems"

In a recent paper [4], Duarte and Jutten investigated the Blind Source Separation (BSS) problem, for the nonlinear mixing model that they introduced in that paper. They proposed to solve this problem by using information-theoretic tools, more precisely by minimizing the mutual information (MI) of the outputs of the separating structure. When applying the MI approach to BSS problems, one usually determines the analytical expressions of the derivatives of the MI with respect to the parameters of the considered separating model. In the literature, these calculations were mainly reported for linear mixtures up to now. They are more complex for nonlinear mixtures, due to dependencies between the considered quantities. Moreover, the notations commonly employed by the BSS community in such calculations may become misleading when using them for nonlinear mixtures, due to the above-mentioned dependencies. We claim that the calculations reported in [4] contain an error, because they did not take into account all these dependencies. In this document, we therefore explain this phenomenon, by showing the effect of indirect dependencies on the application of the MI approach to the mixing and separating models considered in [4]. We thus introduce a corrected expression of the gradient of the considered BSS criterion based on MI. This correct gradient may then e.g. be used to optimize the adaptive coefficients of the considered separating system by means of the well-known gradient descent algorithm. As explained hereafter, this investigation has some similarities with an analysis that we previously reported in another arXiv document [3]. However, these two investigations concern different problems (mixture and separating structure, mathematical tools: see paper).

💡 Research Summary

The paper revisits the mutual‑information (MI) minimization approach proposed by Duarte and Jutten for blind source separation (BSS) of nonlinear recurrent mixtures. While the original work suggested that the separating system’s parameters could be optimized by directly differentiating the MI of the outputs, the present analysis demonstrates that this differentiation was incomplete. In nonlinear recurrent structures the output at any time depends not only on the current parameters but also on previous outputs, which themselves depend on the parameters. Consequently, indirect dependencies generate additional terms in the Jacobian that were omitted in the original derivation.

The authors first restate the mathematical model: source signals s are mixed by a nonlinear function f to produce observations x = f(s); a separating system g, parameterized by θ, processes x recursively, yielding outputs y = g(x;θ) where each y_t may depend on y_{t‑1} and x_t. The MI cost is I(y) = H(y) – Σ_i H(y_i|y_{‑i}), and the gradient ∇_θ I must be obtained by applying the chain rule through the entire computational graph. The missing terms correspond to products of partial derivatives of later outputs with respect to earlier outputs and then with respect to θ.

A corrected gradient expression is derived: ∇_θ I = E