The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units

This paper explores the relationship between the condition number of a neural network’s weight tensor and the extent of information encoded by the associated processing unit, viewed through the lens of information theory. It argues that a high condition number, though not sufficient for effective knowledge encoding, may indicate that the unit has learned to selectively amplify and compress information. This intuition is formalized for linear units with Gaussian inputs, linking the condition number and the transformation’s log-volume scaling factor to the characteristics of the output entropy and the geometric properties of the learned transformation. The analysis demonstrates that for a fixed weight norm, a concentrated distribution of singular values (high condition number) corresponds to reduced overall information transfer, indicating a specialized and efficient encoding strategy. Furthermore, the linear stage entropy bound provides an upper limit on post-activation information for contractive, element-wise nonlinearities, supporting the condition number as a scale-invariant proxy for encoding capacity in practical neural networks. An empirical case study applies these principles to guide selective fine-tuning of Large Language Models for both a new task and a new input modality. The experiments show that the proposed method, named KappaTune, effectively mitigates catastrophic forgetting. Unlike many existing catastrophic forgetting mitigation methods that rely on access to pre-training statistics, which are often unavailable, this selective fine-tuning approach offers a way to bypass this common requirement.

💡 Research Summary

This paper investigates the relationship between the condition number of a neural network’s weight tensor and the extent of information encoded by the corresponding processing unit, proposing the condition number as a scale-invariant proxy for encoding significance. The core argument is that a high condition number, while not a sufficient condition for effective encoding, may indicate that a unit has learned to selectively amplify and compress information along specific directions, leading to efficient and specialized knowledge representation.

The theoretical analysis formalizes this intuition for linear units with Gaussian inputs. It demonstrates that the differential entropy of the output is directly related to the product of the singular values of the weight matrix. A key theorem proves that under a fixed Frobenius norm constraint, the output entropy is maximized when all singular values are equal, i.e., when the condition number κ is 1. This means a κ≈1 transformation has the highest potential for information transfer (or uncertainty preservation). Conversely, for a fixed norm, a concentrated singular value distribution (high κ) results in a smaller log-volume scaling factor, reducing overall information transfer. In a well-trained model, this anisotropy reflects an efficient encoding strategy where the unit focuses its representational capacity on task-relevant, discriminative features while attenuating less predictive variations, aligning with the Information Bottleneck principle. The analysis is extended to nonlinear units, showing that for common contractive activation functions, the linear-stage entropy serves as an upper bound for the post-activation information content.

Building on this theoretical foundation, the paper introduces a practical application: the KappaTune algorithm for selective fine-tuning to mitigate catastrophic forgetting. The guiding hypothesis is that parameters with low condition numbers (κ≈1) are less specialized, have higher information transfer potential, and are thus more adaptable to new tasks without disrupting existing knowledge. In contrast, parameters with high condition numbers (κ≫1) are highly specialized and their modification is likely to cause forgetting. KappaTune operates by computing the condition number for all eligible weight tensors in a pre-trained model, then unfreezing and fine-tuning only a small budget of parameters with the smallest condition numbers, while keeping the vast majority frozen.

The method is evaluated through a case study on adapting a Large Language Model to a new task and a new input modality (audio). The experiments show that KappaTune effectively mitigates catastrophic forgetting, enabling successful adaptation while preserving prior knowledge. A significant practical advantage of this approach is that it operates based solely on intrinsic model properties (the weight tensors themselves), requiring no access to pre-training data or task-specific statistics, which are often unavailable in real-world scenarios. Thus, the paper bridges theoretical insights from linear algebra and information theory to a novel, practical algorithm for robust continual learning.