We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, one does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares estimator, i.e., they achieve lower mean-squared error for any value of the parameter vector. Both Stein's estimator and its positive-part correction can be derived within the blind minimax framework. Furthermore, our approach can be readily extended to a wider class of estimation problems than Stein's estimator, which is defined only for white noise and non-transformed measurements. We show through simulations that the BMEs generally outperform previous extensions of Stein's technique.
Deep Dive into Blind Minimax Estimation.
We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, one does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares estimator, i.e., they achieve lower mean-squared error for any value of the parameter vector. Both Stein’s estimator and its positive-part correction can be derived within the blind minimax framework. Furthermore, our approach can be readily extended to a wider class of estimation problems than Stein’s estimator, which is defined only for white noise and non-transformed measurements. We show through simulations that the BMEs generally outperform p
The problem of estimating a parameter vector from noisy measurements has countless applications in science and engineering. Such estimation problems are typically modeled either in a Bayesian setting, in which a prior distribution on the parameter is assumed, or in a deterministic setting, in which no prior is assumed [1]. This paper examines the deterministic estimation problem. We further assume that the measurements y = Hx + w are linear combinations of the parameter vector x, to which Gaussian noise w is added. Here the transformation matrix H and the noise covariance are assumed to be known. We seek an estimate x which approximates x in the sense of minimal mean-squared error (MSE).
This ubiquitous problem was first addressed by Gauss [2] and Legendre [3], who proposed the classical least-squares (LS) estimator. Several lines of reasoning can be used to support the LS approach. One argument is that the LS estimator minimizes the squared error between the measurements y and the transformed estimate ŷ = Hx. The LS estimator is also the maximum likelihood solution for Gaussian noise. However, neither of these criteria are directly related to the MSE, or to any other measure of the distance between x and x. Another property of the LS solution is that it is the unbiased estimator achieving minimal MSE. Yet by removing the requirement of unbiasedness, estimators yielding lower MSE can be constructed. While linearity and unbiasedness may be intuitively appealing properties, they have no relation to the primary goal at hand, namely, achieving low estimation error. Indeed, there are many examples in which the requirement of unbiasedness results in absurd estimators [4].
Because the parameter vector x is deterministic, the MSE E xx 2 is generally a function of x. In other words, one method may be better than another for some values of x, and worse for other values. For instance, the trivial estimator x = 0 achieves optimal MSE when x = 0, but its performance is otherwise poor. Nonetheless, it is possible to impose a partial order among estimation techniques [5], as follows. An estimator x1 is said to strictly dominate a different estimator x2 if the MSE of x1 is lower than that of x2 , for all values of x. If the MSE of x1 is never higher than that of x2 , and is strictly lower for at least one parameter value, then x1 is said to dominate x2 . An estimator is said to be admissible if it is not dominated by any other estimator. Surprisingly, when the parameter vector contains three or more elements, the LS method turns out to be inadmissible, i.e., some techniques always achieve lower MSE [6]. Thus, it is of interest to characterize the class of admissible estimators, and to find techniques which dominate LS.
The study of admissibility is sometimes restricted to linear methods x = Gy. A linear admissible estimator is one which is not dominated by any other linear strategy. A simple rule characterizes the class of linear admissible techniques [7], and, given any linear inadmissible estimator, it is possible to construct a linear admissible alternative which dominates it [8]. However, the problem of admissibility is considerably more intricate when the linearity restriction is removed; generally, admissible estimators are either trivial (e.g., x = 0) or exceedingly complex [9], [10]. As a result, much research has focused on finding simple nonlinear techniques which dominate LS.
Early work on LS-dominating strategies considered the independent, identical-distribution (i.i.d.) case, for which H = I and the noise is white. Among these, the James-Stein estimator [5], [11] is the best-known example; others approaches include the works of Stein [6] and Thompson [12]. Various “extended” James-Stein methods were later constructed for the general (non-i.i.d.) case [13]- [16]. Of these, Bock’s technique [13] is quoted most often [16], [17]. However, none of these approaches has become a standard alternative to the LS estimator, and they are rarely used in practice in engineering applications [16]. Perhaps one reason for this is that some of the estimators are poorly justified and seem counterintuitive, and as such they are sometimes regarded with skepticism (see discussion following [18]). Another reason is that many of these approaches (including Bock’s method) result in shrinkage estimators, consisting of a gain factor multiplying the LS estimate. Shrinkage techniques can certainly be used to reduce MSE; however, in the non-i.i.d. case, some measurements are noisier than others, and thus a single shrinkage factor for all measurements can be considered suboptimal. Furthermore, in some applications, a gain factor has no effect on final system performance: for example, in an image reconstruction problem, multiplying the entire image by a constant does not improve quality.
In this paper, we provide a framework for generating a wide class of low-complexity, LS-dominating estimators, which are constructed from a simple, in
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
