We consider median regression and, more generally, a possibly infinite collection of quantile regressions in high-dimensional sparse models. In these models the overall number of regressors $p$ is very large, possibly larger than the sample size $n$, but only $s$ of these regressors have non-zero impact on the conditional quantile of the response variable, where $s$ grows slower than $n$. We consider quantile regression penalized by the $\ell_1$-norm of coefficients ($\ell_1$-QR). First, we show that $\ell_1$-QR is consistent at the rate $\sqrt{s/n} \sqrt{\log p}$. The overall number of regressors $p$ affects the rate only through the $\log p$ factor, thus allowing nearly exponential growth in the number of zero-impact regressors. The rate result holds under relatively weak conditions, requiring that $s/n$ converges to zero at a super-logarithmic speed and that regularization parameter satisfies certain theoretical constraints. Second, we propose a pivotal, data-driven choice of the regularization parameter and show that it satisfies these theoretical constraints. Third, we show that $\ell_1$-QR correctly selects the true minimal model as a valid submodel, when the non-zero coefficients of the true model are well separated from zero. We also show that the number of non-zero coefficients in $\ell_1$-QR is of same stochastic order as $s$. Fourth, we analyze the rate of convergence of a two-step estimator that applies ordinary quantile regression to the selected model. Fifth, we evaluate the performance of $\ell_1$-QR in a Monte-Carlo experiment, and illustrate its use on an international economic growth application.
Deep Dive into L1-Penalized Quantile Regression in High-Dimensional Sparse Models.
We consider median regression and, more generally, a possibly infinite collection of quantile regressions in high-dimensional sparse models. In these models the overall number of regressors $p$ is very large, possibly larger than the sample size $n$, but only $s$ of these regressors have non-zero impact on the conditional quantile of the response variable, where $s$ grows slower than $n$. We consider quantile regression penalized by the $\ell_1$-norm of coefficients ($\ell_1$-QR). First, we show that $\ell_1$-QR is consistent at the rate $\sqrt{s/n} \sqrt{\log p}$. The overall number of regressors $p$ affects the rate only through the $\log p$ factor, thus allowing nearly exponential growth in the number of zero-impact regressors. The rate result holds under relatively weak conditions, requiring that $s/n$ converges to zero at a super-logarithmic speed and that regularization parameter satisfies certain theoretical constraints. Second, we propose a pivotal, data-driven choice of the re
1. Introduction. Quantile regression is an important statistical method for analyzing the impact of regressors on the conditional distribution of a response variable (cf. [27], [24]). It captures the heterogeneous impact of regressors on different parts of the distribution [8], exhibits robustness to outliers [22], has excellent computational properties [34], and has wide applicability [22]. The asymptotic theory for quantile regression has been developed under both a fixed number of regressors and an increasing number of regressors. The asymptotic theory under a fixed number of regressors is given in [24], [33], [18], [20], [13] and others. The asymptotic theory under an increasing number of regressors is given in [19] and [3,4], covering the case where the number of regressors p is negligible relative to the sample size n (i.e., p = o(n)).
In this paper, we consider quantile regression in high-dimensional sparse models (HDSMs). In such models, the overall number of regressors p is very large, possibly much larger than the sample size n. However, the number of significant regressors for each conditional quantile of interest is at most s, which is smaller than the sample size, that is, s = o(n). HDSMs ( [7,12,32]) have emerged to deal with many new applications arising in biometrics, signal processing, machine learning, econometrics, and other areas of data analysis where high-dimensional data sets have become widely available.
A number of papers have begun to investigate estimation of HDSMs, focusing primarily on penalized mean regression, with the โ 1 -norm acting as a penalty function [7,12,26,32,39,41]. [7,12,26,32,41] demonstrated the fundamental result that โ 1 -penalized least squares estimators achieve the rate s/n โ log p, which is very close to the oracle rate s/n achievable when the true model is known. [39] demonstrated a similar fundamental result on the excess forecasting error loss under both quadratic and non-quadratic loss functions. Thus the estimator can be consistent and can have excellent forecasting performance even under very rapid, nearly exponential, growth of the total number of regressors p. See [7, 9-11, 15, 30, 35] for many other interesting developments and a detailed review of the existing literature. Our paper’s contribution is to develop a set of results on model selection and rates of convergence for quantile regression within the HDSM framework. Since ordinary quantile regression is inconsistent in HDSMs, we consider quantile regression penalized by the โ 1 -norm of parameter coefficients, denoted โ 1 -QR. First, we show that under general conditions โ 1 -QR estimates of regression coefficients and regression functions are consistent at the near-oracle rate s/n log(p โจ n), uniformly in a compact interval U โ (0, 1) of quantile indices. 1 (This result is different from and hence complementary to [39]’s fundamental results on the rates for excess forecasting error loss.) Second, in order to make โ 1 -QR practical, we propose a partly pivotal, data-driven choice of the penalty level, and show that this choice leads to the same sharp convergence rate. Third, we show that โ 1 -QR correctly selects the true model as a valid submodel when the non-zero coefficients of the true model are well separated from zero. Fourth, we also propose and analyze the post-penalized estimator (post-โ 1 -QR), which applies ordi-nary, unpenalized quantile regression to the model selected by the penalized estimator, and thus aims at reducing the regularization bias of the penalized estimator. We show that under similar conditions post-โ 1 -QR can perform as well as โ 1 -QR in terms of the rate of convergence, uniformly over U , even if the โ 1 -QR-based model selection misses some components of the true models. This occurs because โ 1 -QR-based model selection only misses those components that have relatively small coefficients. Moreover, post-โ 1 -QR can perform better than โ 1 -QR if the โ 1 -QR-based model selection correctly includes all components of the true model as a subset. (Obviously, post-โ 1 -QR can perform as well as the oracle if the โ 1 -QR perfectly selects the true model, which is, however, unrealistic for many designs of interest.) Fifth, we illustrate the use of โ 1 -QR and post-โ 1 -QR with a Monte Carlo experiment and an international economic growth example. To the best of our knowledge, all of the above results are new and contribute to the literature on HDSMs. Our results on post-penalized estimators and some proof techniques could also be of interest in other problems. We provide further technical comparisons to the literature in Section 2.
In what follows, we implicitly index all parameter values by the sample size n, but we omit the index whenever this does not cause confusion. We use the empirical process notation as defined in [40]. In particular, given a random sample Z 1 , …, Z n , let G n (f ) =
) and E n f = E n f (Z i ) := n i=1 f (Z i )/n. We use the notation
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
