Compressed Regression

The approach we develop here compresses the data by a random linear or affine transformation, reducing the number of data records exponentially, while preserving the number of original input variables. These compressed data can then be made available for statistical analyses; we focus on the problem of sparse linear regression for high dimensional data. Informally, our theory ensures that the relevant predictors can be learned from the compressed data as well as they could be from the original uncompressed data. Moreover, the actual predictions based on new examples are as accurate as they would be had the original data been made available. However, the original data are not recoverable from the compressed data, and the compressed data effectively reveal no more information than would be revealed by a completely new sample. At the same time, the inference algorithms run faster and require fewer resources than the much larger uncompressed data would require. In fact, the original data need never be stored; they can be transformed “on the fly” as they come in.

In more detail, the data are represented as a n × p matrix X . Each of the p columns is an attribute, and each of the n rows is the vector of attributes for an individual record. The data are compressed by a random linear transformation

where is a random m × n matrix with m ≪ n. It is also natural to consider a random affine transformation

where is a random m × p matrix. Such transformations have been called “matrix masking” in the privacy literature (Duncan and Pearson, 1991). The entries of and are taken to be independent Gaussian random variables, but other distributions are possible. We think of X as “public,” while and are private and only needed at the time of compression. However, even with = 0 and known, recovering X from X requires solving a highly under-determined linear system and comes with information theoretic privacy guarantees, as we demonstrate.

In standard regression, a response Y = Xβ + ǫ ∈ R n is associated with the input variables, where ǫ i are independent, mean zero additive noise variables. In compressed regression, we assume that the response is also compressed, resulting in the transformed response Y ∈ R m given by

Note that under compression, the transformed noise ǫ = ǫ is not independent across examples.

In the sparse setting, the parameter vector β ∈ R p is sparse, with a relatively small number s of nonzero coefficients supp(β) = j : β j = 0 . Two key tasks are to identify the relevant variables, and to predict the response x T β for a new input vector x ∈ R p . The method we focus on is ℓ 1regularized least squares, also known as the lasso (Tibshirani, 1996). The main contributions of this paper are two technical results on the performance of this estimator, and an informationtheoretic analysis of the privacy properties of the procedure. Our first result shows that the lasso is sparsistent under compression, meaning that the correct sparse set of relevant variables is identified asymptotically. Omitting details and technical assumptions for clarity, our result is the following. then the compressed lasso solution

includes the correct variables, asymptotically:

(1.9)

Our second result shows that the lasso is persistent under compression. Roughly speaking, persistence (Greenshtein and Ritov, 2004) means that the procedure predicts well, as measured by the predictive risk (1.10) where now X ∈ R p is a new input vector and Y is the associated response. Persistence is a weaker condition than sparsistency, and in particular does not assume that the true model is linear.

Persistence (Theorem 4.1): Given a sequence of sets of estimators B n,m , the sequence of compressed lasso estimators

β n,m = arg min

is persistent with the oracle risk over uncompressed data with respect to B n,m , meaning that

R(β) P -→ 0, as n → ∞.

(1.12) in case log 2 (np) ≤ m ≤ n and the radius of the ℓ 1 ball satisfies L n,m = o (m/ log(np)) 1/4 .

Our third result analyzes the privacy properties of compressed regression. We consider the problem of recovering the uncompressed data X from the compressed data X = X + . To preserve privacy, the random matrices and should remain private. However, even in the case where = 0 and is known, if m ≪ min(n, p) the linear system X = X is highly underdetermined. We evaluate privacy in information theoretic terms by bounding the average mutual information I ( X ; X )/np per matrix entry in the original data matrix X , which can be viewed as a communication rate. Bounding this mutual information is intimately connected with the problem of computing the channel capacity of certain multiple-antenna wireless communication systems (Marzetta and Hochwald, 1999;Telatar, 1999).

Information Resistence (Propositions 5.1 and 5.2): The rate at which information about X is revealed by the compressed data X satisfies

where the supremum is over distributions on the original data X .

As summarized by these results, compressed regression is a practical procedure for sparse learning in high dimensional data that has provably good properties. This basic technique has connections in the privacy literature with matrix masking and other methods, yet most of the existing work in this direction has been heuristic and without theoretical guarantees; connections with this literature are briefly reviewed in Section 2.C. Compressed regression builds on the ideas underlying compressed sensing and sparse inference in high dimensional data, topics which have attracted a great deal of recent interest in the statistics and signal processing communities; the connections with this literature are reviewed in Section 2.B and 2.A.

The remainder of the paper is organized as follows. In Section 2 we review relevant work from high dimensional statistical inference, compressed sensing and privacy. Section 3 presents our analysis of the sparsistency properties of the compressed lasso. Our approach follows the methods introduced by Wainwright (2006) in the uncompressed case. Section 4 proves that compressed regression is persistent. Section 5 derives upper bounds on the mutual information between the compressed data X and the uncompressed data X , after identifying a correspondence with the problem of computing channel capacity for a certain model of a multiple-antenna mobile communication channel. Section 6 includes the results of experimental simulations, showing that the empirical performance of the compressed lasso is consistent with our theoretical analysis. We evaluate the ability of the procedure to recover the relevant variables (sparsistency) and to predict well (persistence). The technical details of the proof of sparsistency are collected at the end of the paper, in Section 7.B. The paper concludes with a discussion of the results and directions for future work in Section 8.

In this section we briefly review relevant related work in high dimensional statistical inference, compressed sensing, and privacy, to place our work in context.

We adopt standard notation where a data matrix X has p variables and n records; in a linear model the response Y = Xβ + ǫ ∈ R n is thus an n-vector, and the noise ǫ i is independent and mean zero, E(ǫ) = 0. The usual estimator of β is the least squares estimator β = (X T X ) -1 X T Y.

(2.1)

However, this estimator has very large variance when p is large, and is not even defined when p > n. An estimator that has received much attention in the recent literature is the lasso β n (Tibshirani, 1996), defined as

where λ n is a regularization parameter. The practical success and importance of the lasso can be attributed to the fact that in many cases β is sparse, that is, it has few large components. For example, data are often collected with many variables in the hope that at least a few will be useful for prediction. The result is that many covariates contribute little to the prediction of Y , although it is not known in advance which variables are important. Recent work has greatly clarified the properties of the lasso estimator in the high dimensional setting.

One of the most basic desirable properties of an estimator is consisistency; an estimator β n is consistent in case

(2.4) Meinshausen and Yu (2006) have recently shown that the lasso is consistent in the high dimensional setting. If the underlying model is sparse, a natural yet more demanding criterion is to ask that the estimator correctly identify the relevant variables. This may be useful for interpretation, dimension reduction and prediction. For example, if an effective procedure for high-dimensional data can be used to identify the relevant variables in the model, then these variables can be isolated and their coefficients estimated by a separate procedure that works well for low-dimensional data.

An estimator is sparsistent1 if

where supp(β) = {j : j = 0}. Asymptotically, a sparsistent estimator has nonzero coefficients only for the true relevant variables. Sparsistency proofs for high dimensional problems have appeared recently in a number of settings. Meinshausen and Buhlmann (2006) consider the problem of estimating the graph underlying a sparse Gaussian graphical model by showing sparsistency of the lasso with exponential rates of convergence on the probability of error. Zhao and Yu (2007) show sparsistency of the lasso under more general noise distributions. Wainwright (2006) characterizes the sparsistency properties of the lasso by showing that there is a threshold sample size n( p, s) above which the relevant variables are identified, and below which the relevant variables fail to be identified, where s = β 0 is the number of relevant variables. More precisely, Wainwright (2006) shows that when X comes from a Gaussian ensemble, there exist fixed constants 0 < θ ℓ ≤ 1 and 1 ≤ θ u < +∞, where θ ℓ = θ u = 1 when each row of X is chosen as an independent Gaussian random vector ∼ N (0, I p× p ), then for any ν > 0, if

then the lasso identifies the true variables with probability approaching one. Conversely, if

then the probability of recovering the true variables using the lasso approaches zero. These results require certain incoherence assumptions on the data X ; intuitively, it is required that an irrelevant variable cannot be too strongly correlated with the set of relevant variables. This result and Wainwright’s method of analysis are particularly relevant to the current paper; the details will be described in the following section. In particular, we refer to this result as the Gaussian Ensemble result. However, it is important to point out that under compression, the noise ǫ = ǫ is not independent. This prevents one from simply applying the Gaussian Ensemble results to the compressed case. Related work that studies information theoretic limits of sparsity recovery, where the particular estimator is not specified, includes (Wainwright, 2007;Donoho and Tanner, 2006). Sparsistency in the classification setting, with exponential rates of convergence for ℓ 1 -regularized logistic regression, is studied by Wainwright et al. (2007).

An alternative goal is accurate prediction. In high dimensions it is essential to regularize the model in some fashion in order to control the variance of the estimator and attain good predictive risk.

Persistence for the lasso was first defined and studied by Greenshtein and Ritov (2004). Given a sequence of sets of estimators B n , the sequence of estimators

where

Thus, a sequence of estimators is persistent if it asymptotically predicts as well as the oracle within the class, which minimizes the population risk; it can be achieved under weaker assumptions than are required for sparsistence. In particular, persistence does not assume the true model is linear, and it does not require strong incoherence assumptions on the data. The results of the current paper show that sparsistence and persistence are preserved under compression.

Compressed regression has close connections to, and draws motivation from, compressed sensing (Donoho, 2006;Candès et al., 2006;Candès and Tao, 2006;Rauhut et al., 2007). However, in a sense, our motivation here is the opposite to that of compressed sensing. While compressed sensing of X allows a sparse X to be reconstructed from a small number of random measurements, our goal is to reconstruct a sparse function of X . Indeed, from the point of view of privacy, approximately reconstructing X , which compressed sensing shows is possible if X is sparse, should be viewed as undesirable; we return to this point in Section 5.

Several authors have considered variations on compressed sensing for statistical signal processing tasks (Duarte et al., 2006;Davenport et al., 2006;Haupt et al., 2006;Davenport et al., 2007). The focus of this work is to consider certain hypothesis testing problems under sparse random measurements, and a generalization to classification of a signal into two or more classes. Here one observes y = x, where y ∈ R m , x ∈ R n and is a known random measurement matrix. The problem is to select between the hypotheses

where ǫ ∈ R n is additive Gaussian noise. Importantly, the setup exploits the “universality” of the matrix , which is not selected with knowledge of s i . The proof techniques use concentration properties of random projection, which underlie the celebrated lemma of Johnson and Lindenstrauss (1984). The compressed regression problem we introduce can be considered as a more challenging statistical inference task, where the problem is to select from an exponentially large set of linear models, each with a certain set of relevant variables with unknown parameters, or to predict as well as the best linear model in some class. Moreover, a key motivation for compressed regression is privacy; if privacy is not a concern, simple subsampling of the data matrix could be an effective compression procedure.

Research on privacy in statistical data analysis has a long history, going back at least to Dalenius (1977a); we refer to Duncan and Pearson (1991) for discussion and further pointers into this literature. The compression method we employ has been called matrix masking in the privacy literature. In the general method, the n × p data matrix X is transformed by pre-multiplication, post-multiplication, and addition into a new m × q matrix

(2.10)

The transformation A operates on data records for fixed covariates, and the transformation B operates on covariates for a fixed record. The method encapsulated in this transformation is quite general, and allows the possibility of deleting records, suppressing subsets of variables, data swapping, and including simulated data. In our use of matrix masking, we transform the data by replacing each variable with a relatively small number of random averages of the instances of that variable in the data. In other work, Sanil et al. (2004) consider the problem of privacy preserving regression analysis in distributed data, where different variables appear in different databases but it is of interest to integrate data across databases. The recent work of Ting et al. (2007) considers random orthogonal mappings X → R X = X where R is a random rotation (rank n), designed to preserve the sufficient statistics of a multivariate Gaussian and therefore allow regression estimation, for instance. This use of matrix masking does not share the information theoretic guarantees we present in Section 5. We are not aware of previous work that analyzes the asymptotic properties of a statistical estimator under matrix masking in the high dimensional setting.

The work of Liu et al. (2006) is closely related to the current paper at a high level, in that it considers low rank random linear transformations of either the row space or column space of the data X . Liu et al. (2006) note the Johnson-Lindenstrauss lemma, which implies that ℓ 2 norms are approximately preserved under random projection, and argue heuristically that data mining procedures that exploit correlations or pairwise distances in the data, such as principal components analysis and clustering, are just as effective under random projection. The privacy analysis is restricted to observing that recovering X from X requires solving an under-determined linear system, and arguing that this prevents the exact values from being recovered.

An information-theoretic quantification of privacy was formulated by Agrawal and Aggarwal (2001). Given a random variable X and a transformed variable X , Agrawal and Aggarwal (2001) define the conditional privacy loss of X given X as

which is simply a transformed measure of the mutual information between the two random variables. In our work we identify privacy with the rate of information communicated about X through X under matrix masking, maximizing over all distributions on X . We furthermore identify this with the problem of computing, or bounding, the Shannon capacity of a multi-antenna wireless communication channel, as modeled by Telatar (1999) and Marzetta and Hochwald (1999).

Finally, it is important to mention the extensive and currently active line of work on cryptographic approaches to privacy, which have come mainly from the theoretical computer science community.

For instance, Feigenbaum et al. (2006) develop a framework for secure computation of approximations; intuitively, a private approximation of a function f is an approximation f that does not reveal information about x other than what can be deduced from f (x). Indyk and Woodruff (2006) consider the problem of computing private approximate nearest neighbors in this setting. Dwork (2006) revisits the notion of privacy formulated by Dalenius (1977b), which intuitively demands that nothing can be learned about an individual record in a database that cannot be learned without access to the database. An impossibility result is given which shows that, appropriately formalized, this strong notion of privacy cannot be achieved. An alternative notion of differential privacy is proposed, which allows the probability of a disclosure of private information to change by only a small multiplicative factor, depending on whether or not an individual participates in the database. This line of work has recently been built upon by Dwork et al. (2007), with connections to compressed sensing, showing that any method that gives accurate answers to a large fraction of randomly generated subset sum queries must violate privacy.

In the standard setting, X is a n × p matrix, Y = Xβ + ǫ is a vector of noisy observations under a linear model, and p is considered to be a constant. In the high-dimensional setting we allow p to grow with n. The lasso refers to the following quadratic program:

In Lagrangian form, this becomes the optimization problem

where the scaling factor 1/2n is chosen by convention and convenience. For an appropriate choice of the regularization parameter λ = λ(Y, L), the solutions of these two problems coincide.

In compressed regression we project each column X j ∈ R n of X to a subspace of m dimensions, using an m × n random projection matrix . We shall assume that the entries of are independent Gaussian random variables:

Let X = X be the compressed matrix of covariates, and let Y = Y be the compressed response.

Our objective is to estimate β in order to determine the relevant variables, or to predict well. The compressed lasso is the optimization problem, for Y = Xβ + ǫ = X + ǫ:

with m being the set of optimal solutions:

Thus, the transformed noise ǫ is no longer i.i.d., a fact that complicates the analysis. It is convenient to formalize the model selection problem using the following definitions.

Definition 3.1. (Sign Consistency) A set of estimators n is sign consistent with the true β if

where sgn(•) is given by sgn

As a shorthand, we use

to denote the event that a sign consistent solution exists.

The lasso objective function is convex in β, and strictly convex for p ≤ n. Therefore the set of solutions to the lasso and compressed lasso (3.4) is convex: if β and β ′ are two solutions, then by convexity β + ρ( β ′β) is also a solution for any ρ ∈ [0, 1].

Definition 3.2. (Sparsistency) A set of estimators n is sparsistent with the true β if

Clearly, if a set of estimators is sign consistent then it is sparsistent. Although sparsistency is the primary goal in selecting the correct variables, our analysis establishes conditions for the slightly stronger property of sign consistency.

All recent work establishing results on sparsity recovery assumes some form of incoherence condition on the data matrix X . Such a condition ensures that the irrelevant variables are not too strongly correlated with the relevant variables. Intuitively, without such a condition the lasso may be subject to false positives and negatives, where an relevant variable is replaced by a highly correlated relevant variable. To formulate such a condition, it is convenient to introduce an additional piece of notation. Let S = {j : β j = 0} be the set of relevant variables and let S c = {1, . . . , p} \ S be the set of irrelevant variables. Then X S and X S c denote the corresponding sets of columns of the matrix X . We will impose the following incoherence condition; related conditions are used by Donoho et al. (2006) and Tropp (2004) in a deterministic setting.

Definition 3.3. (S-Incoherence) Let X be an n × p matrix and let S ⊂ {1, . . . , p} be nonempty. We say that X is S-incoherent in case

where

Although it is not explicitly required, we only apply this definition to X such that columns of X satisfy X j 2 2 = (n), ∀ j ∈ {1, . . . , p}. We can now state the main result of this section.

Theorem 3.4. Suppose that, before compression, we have Y = Xβ * + ǫ, where each column of X is normalized to have ℓ 2 -norm n, and ε ∼ N (0, σ 2 I n ). Assume that X is S-incoherent, where S = supp(β * ), and define s = |S| and ρ m = min i∈S |β * i |. We observe, after compression, (3.11) where Y = Y , X = X , and ǫ = ǫ, where i j ∼ N (0, 1/n). Suppose

Then the compressed lasso is sparsistent:

where β m is an optimal solution to (3.4).

Our overall approach is to follow a deterministic analysis, in the sense that we analyze X as a realization from the distribution of from a Gaussian ensemble. Assuming that X satisfies the S-incoherence condition, we show that with high probability X also satisfies the S-incoherence condition, and hence the incoherence conditions (7.1a) and (7.1b) used by Wainwright (2006). In addition, we make use of a large deviation result that shows T is concentrated around its mean I m×m , which is crucial for the recovery of the true sparsity pattern. It is important to note that the compressed noise ǫ is not independent and identically distributed, even when conditioned on .

In more detail, we first show that with high probability 1n -c for some c ≥ 2, the projected data X satisfies the following properties:

2. X is S-incoherent, and also satisfies the incoherence conditions (7.1a) and (7.1b).

In addition, the projections satisfy the following properties:

1. Each entry of T -I is at most b log n/n for some constant b, with high probability;

These facts allow us to condition on a “good” and incoherent X , and to proceed as in the deterministic setting with Gaussian noise. Our analysis then follows that of Wainwright (2006). Recall S is the set of relevant variables in β and S c = {1, . . . , p} \ S is the set of irrelevant variables. To explain the basic approach, first observe that the KKT conditions imply that β ∈ R p is an optimal solution to (3.4), i.e., β ∈ m , if and only if there exists a subgradient

Hence, the E sgn( β) = sgn(β * ) can be shown to be equivalent to requiring the existence of a solution β ∈ R p such that sgn( β) = sgn(β * ), and a subgradient z ∈ ∂ β 1 , such that the following equations hold:

where z S = sgn(β * S ) and | z S c | ≤ 1 by definition of z. The existence of solutions to equations (3.17a) and (3.17b) can be characterized in terms of two events E(V ) and E(U ). The proof proceeds by showing that P(E(V )) → 1 and P(E(U )) → 1 as m → ∞.

In the remainder of this section we present the main steps of the proof, relegating the technical details to Section 7.B. To avoid unnecessary clutter in notation, we will use Z to denote the compressed data X = X and W to denote the compressed response Y = Y , and ω = ǫ to denote the compressed noise.

In order for the estimated β m to be close to the solution of the uncompressed lasso, we require the stability of inner products of columns of X under multiplication with the random matrix , in the sense that

(3.18)

Toward this end we have the following result, adapted from Rauhut et al. (2007), where for each entry in , the variance is 1 m instead of 1 n .

Assume that is an m × n random matrix with independent N (0, n -1 ) entries (independent of x, y). Then for all τ > 0

We next summarize the properties of X that we require. The following result implies that, with high probability, incoherence is preserved under random projection.

Proposition 3.6. Let X be a (deterministic) design matrix that is S-incoherent with ℓ 2 -norm n, and let be a m × n random matrix with independent N (0, n -1 ) entries. Suppose that

for some c ≥ 2, where C 1 , C 2 are defined in Lemma 3.5. Then with probability at least 1 -1/n c the following properties hold for Z = X :

1. Z is S-incoherent; in particular:

2. Z = X is incoherent in the sense of (7.1a) and (7.1b):

3. The ℓ 2 norm of each column is approximately preserved, for all j :

Finally, we have the following large deviation result for the projection matrix , which guarantees that R = T -I m×m is small entrywise.

C. Proof of Theorem 3.4

We first state necessary and sufficient conditions on the event E(sgn( β m ) = sgn(β * )). Note that this is essentially equivalent to Lemma 1 in Wainwright (2006); a proof of this lemma is included in Section 7.F for completeness.

Lemma 3.8. Assume that the matrix Z T S Z S is invertible. Then for any given λ m > 0 and noise vector ω ∈ R m , E sgn( β m ) = sgn(β * ) holds if and only if the following two conditions hold:

) and e i ∈ R s be the vector with 1 in i th position, and zeros elsewhere; hence e i 2 = 1. Our proof of Theorem 3.4 follows that of Wainwright (2006). We first define a set of random variables that are relevant to (3.25a) and (3.25b):

We first define a set of random variables that are relevant to Condition (3.25a), which holds if and if only the event

holds. For Condition (3.25b), the event

where ρ m := min i∈S |β * i |, is sufficient to guarantee that Condition (3.25b) holds. Now, in the proof of Theorem 3.4, we assume that has been fixed, and Z = X and T behave nicely, in accordance with the results of Section 3.B. Let R = T -I m×m as defined in Theorem 3.7. From here on, we use ( r i, j ) to denote a fixed symmetric matrix with diagonal entries that are 16 log n/n and off-diagonal entries that are 2 log n/n.

We now prove that P (E(V )) and P (E(U )) both converge to one. We begin by stating two technical lemmas that will be required.

Lemma 3.9. (Gaussian Comparison) For any Gaussian random vector (X 1 , . . . , X n ),

Analysis of E(V ). Note that for each V j , for j ∈ S c ,

By Proposition 3.6, we have that

Let us define

from which we obtain

Hence we need to show that

By Markov’s inequality and the Gaussian comparison lemma 3.9, we obtain that

Finally, let us use P = Z S (Z T S Z S ) -1 Z T S = P 2 to represent the projection matrix.

where Z j 2 2 ≤ m + mη 4s by Proposition 3.6, and

given that I -P 2 ≤ 1 and P 2 ≤ 1 and the fact that

Analysis of E(U ). We now show that P (E(U )) → 1. Using the triangle inequality, we obtain the upper bound

The second ℓ ∞ -norm is a fixed value given a deterministic X . Hence we focus on the first norm. We now define, for all i ∈ S, the Gaussian random variable

Given that ǫ ∼ N (0, σ 2 I n×n ), we have for all i ∈ S that

We first bound the first term of (3.44f). By (3.22b), we have that for all i ∈ S, Thus, for all i ∈ S,

We next require the following fact.

Claim 3.11. If m satisfies (3.12), then for all i ∈ S, we have

.

By the Gaussian comparison lemma 3.9, we have

We now apply Markov’s inequality to show that P (E(U )) → 1 due to Condition (3.13b) in the Theorem statement and Lemma 3.10,

which completes the proof.

Persistence ( Greenshtein and Ritov (2004)) is a weaker condition than sparsistency. In particular, we drop the assumption that E(Y |X ) = β T X . Roughly speaking, persistence implies that a procedure predicts well. Let us first review the Greenshtein-Ritov argument; we then adapt it to the compressed case.

Consider a new pair (X, Y ) and suppose we want to predict Y from X . The predictive risk using predictor

Note that this is a well-defined quantity even though we do not assume that E(Y |X ) = β T X . It is convenient to write the risk in the following way. Define Q = (Y, X 1 , . . . , X p ) and denote γ as

Then we can rewrite the risk as

Consider the uncompressed lasso estimator β n which minimizes R n (β) subject to β ∈ B n :

Suppose that, for each j and k,

for every q ≥ 2 and some constants M and s, where

Then, by Bernstein’s inequality,

for some c > 0. Hence, if p n ≤ e n ξ for some 0 ≤ ξ < 1 then

Then, sup

Hence, given a sequence of sets of estimators B n , sup

We claim that under Assumption 1, the sequence of uncompressed lasso procedures as given in (4.7) is persistent, i.e., R(

we thus obtain

(4.17)

For every ǫ > 0, the event R(

Thus, for L n = o((n/ log n) 1/4 ), and for all ǫ > 0

The claim follows from the definition of persistence.

Now we turn to the compressed case. Again we want to predict (X, Y ), but now the estimator β n,m is based on the lasso from the compressed data of dimension m n ; we omit the subscript n from m n wherever we put {n, m} together.

Let γ be as in (4.2) and

Given compressed dimension m n , the original design matrix dimension n and p n , let

Assumption 2. Let Q j denote the j th element of Q. There exists a constant M 1 > 0 such that

Theorem 4.1. Under Assumption 1 and 2, given a sequence of sets of estimators

, the sequence of compressed lasso procedures as in (4.24) is persistent: (4.26) when p n = O e n c for some c < 1/2.

Proof. First note that

We have that sup

(4.28)

We claim that, given

where = 1 n E Q T Q is the same as (4.4), but (4.20) defines the matrix n,m . Hence, given p n = O e n c for some c < 1/2, combining (4.28) and (4.29), we have for given that

Thus for every ǫ > 0, the event R(

The theorem follows from the definition of persistence.

It remains to to show (4.29). We first show the following claim; note that p n = O e n c with c < 1/2 clearly satisfies the condition.

for some chosen constant c 1 and M 1 satisfying Assumption 2, Proof. To see this, let A = (A 1 , . . . , A n ) T denote a generic column vector of Q. Let µ = E(A 2 i ). Under our assumptions, there exists c 1 > 0 such that

where

We have with probability 1 -1/n, that

The claim follows by the union bound for C = 2M 1 .

Thus we assume that Q j 2 2 ≤ Cn for all j , and use the triangle inequality to bound max

where, using p as a shorthand for p n ,

,(4.39a)

.

(4.39b)

We first compare each entry of n,m j k with that of

where

≈ 2.5044 as in Lemma 3.5 and C is defined in Claim 4.2.

Proof. Following arguments that appear before (7.41a), and by Lemma 3.5, it is straight forward to verify:

where C 2 = √ 8e ≈ 7.6885 as in Lemma 3.5. There are at most ( p n +1) p n 2 unique events given that both matrices are symmetric; the claim follows by the union bound.

We have by the union bound and (4.10), (4.38), Claim 4.2, and Claim 4.3,

Hence, given p n = O e n c with c < 1/2, by taking

we have

which completes the proof of the theorem.

Remark 4.4. The main difference between the sequence of compressed lasso estimators and the original uncompressed sequence is that n and m n together define the sequence of estimators for the compressed data. Here m n is allowed to grow from (log 2 (np n )) to n; hence for each fixed n,

defines a subsequence of estimators. In Section 6 we run simulations that compare the empirical risk to the oracle risk on such a subsequence for a fixed n, to illustrate the compressed lasso persistency property.

In this section we derive bounds on the rate at which the compressed data X reveal information about the uncompressed data X . Our general approach is to consider the mapping X → X + as a noisy communication channel, where the channel is characterized by multiplicative noise and additive noise . Since the number of symbols in X is np we normalize by this effective block length to define the information rate r n,m per symbol as

(5.1) Thus, we seek bounds on the capacity of this channel, where several independent blocks are coded. A privacy guarantee is given in terms of bounds on the rate r n,m → 0 decaying to zero. Intuitively, if I (X ; X ) = H (X ) -H (X | X ) ≈ 0, then the compressed data X reveal, on average, no more information about the original data X than could be obtained from an independent sample.

Our analysis yields the rate bound r n,m = O(m/n). Under the lower bounds on m in our sparsistency and persistence analyses, this leads to the information rates

It is important to note, however that these bounds may not be the best possible since they are obtained assuming knowledge of the compression matrix , when in fact the privacy protocol requires that and are not public. Thus, it may be possible to show a faster rate of convergence to zero. We make this simplification since the capacity of the underlying communication channel does not have a closed form, and appears difficult to analyze in general. Conditioning on yields the familiar Gaussian channel in the case of nonzero additive noise .

In the following subsection we first consider the case where additive noise is allowed; this is equivalent to a multiple antenna model in a Rayleigh flat fading environment. While our sparsistency and persistence analysis has only considered = 0, additive noise is expected to give greater privacy guarantees. Thus, extending our regression analysis to this case is an important direction for future work. In Section 5.B we consider the case where = 0 with a direct analysis. This special case does not follow from analysis of the multiple antenna model.

In the multiple antenna model for wireless communication (Marzetta and Hochwald, 1999;Telatar, 1999), there are n transmitter and m receiver antennas in a Raleigh flat-fading environment. The propagation coefficients between pairs of transmitter and receiver antennas are modeled by the matrix entries i j ; they remain constant for a coherence interval of p time periods. Computing the channel capacity over multiple intervals requires optimization of the joint density of pn transmitted signals. Marzetta and Hochwald (1999) prove that the capacity for n > p is equal to the capacity for n = p, and is achieved when X factors as a product of a p × p isotropically distributed unitary matrix and a p × n random matrix that is diagonal, with nonnegative entries. They also show that as p gets large, the capacity approaches the capacity obtained as if the matrix of propagation coefficients were known. Intuitively, this is because the transmitter could send several “training” messages used to estimate , and then send the remaining information based on this estimate.

More formally, the channel is modeled as

where γ > 0, i j ∼ N (0, 1), i j ∼ N (0, 1/n) and 1 n n i=1 E[X 2 i j ] ≤ P, where the latter is a power constraint. The compressed data are then conditionally Gaussian, with

(5.4)

X t j X tl .

(5.5)

Thus the conditional density p(Z | X ) is given by

which completely determines the channel. Note that this distribution does not depend on , and the transmitted signal affects only the variance of the received signal.

The channel capacity is difficult to compute or accurately bound in full generality. However, an upper bound is obtained by assuming that the multiplicative coefficients are known to the receiver. In this case, we have that p(Z , | X ) = p( ) p(Z | , X ), and the mutual information I (Z , ; X ) is given by

(5.7c)

Now, conditioned on , the compressed data Z = X + γ can be viewed as the output of a standard additive noise Gaussian channel. We thus obtain the upper bound sup p(X)

where inequality (5.8c) comes from assuming the p columns of X are independent, and inequality (5.8d) uses Jensen’s inequality and concavity of log det S. Summarizing, we’ve shown the following result.

Proposition 5.1. Suppose that E[X 2 j ] ≤ P and the compressed data are formed by

where is m × n with independent entries i j ∼ N (0, 1/n) and is m × p with independent entries i j ∼ N (0, 1). Then the information rate r n,m satisfies

(5.10)

When = 0, or equivalently γ = 0, the above analysis yields the trivial bound r n,m ≤ ∞. Here we derive a separate bound for this case; the resulting asymptotic order of the information rate is the same, however.

Consider first the case where p = 1, so that there is a single column X in the data matrix. The entries are independently sampled as X i ∼ F where F has mean zero and bounded variance Var(F) ≤ P. Let Z = X ∈ R m . An upper bound on the mutual information I (X ; Z ) again comes from assuming the compression matrix is known. In this case

(5.12)

where the second conditional entropy in (5.11) is zero since Z = X . Now, the conditional variance of

(5.13) Therefore,

where inequality (5.14b) follows from the chain rule and the fact that conditioning reduces entropy, inequality (5.14c) is achieved by taking F = N (0, P), a Gaussian, and inequality (5.14d) uses concavity of log det S. In the case where there are p columns of X , taking each column to be independently sampled from a Gaussian with variance P gives the upper bound

(5.15) Summarizing, we have the following result.

Proposition 5.2. Suppose that E[X 2 j ] ≤ P and the compressed data are formed by

where is m × n with independent entries i j ∼ N (0, 1/n). Then the information rate r n,m satisfies

(5.17)

In this section we report the results of simulations designed to validate the theoretical analysis presented in the previous sections. We first present results that indicate the compressed lasso is comparable to the uncompressed lasso in recovering the sparsity pattern of the true linear model, in accordance with the analysis in Section 3. We then present experimental results on persistence that are in close agreement with the theoretical results of Section 4.

Here we run simulations to compare the compressed lasso with the uncompressed lasso in terms of the probability of success in recovering the sparsity pattern of β * . We use random matrices for both X and , and reproduce the experimental conditions shown in Wainwright (2006). A design parameter is the compression factor f = n m (6.1) which indicates how much the original data are compressed. The results show that when the compression factor f is large enough, the thresholding behaviors as specified in (2.6) and (2.7) for the uncompressed lasso carry over to the compressed lasso, when X is drawn from a Gaussian ensemble. In general, the compression factor f is well below the requirement that we have in Theorem 3.4 in case X is deterministic.

In more detail, we consider the Gaussian ensemble for the projection matrix , where i, j ∼ N (0, 1/n) are independent. The noise vector is always composed of i.i.d. Gaussian random variables ǫ ∼ N (0, σ 2 ), where σ 2 = 1. We consider Gaussian ensembles for the design matrix X with both diagonal and Toeplitz covariance. In the Toeplitz case, the covariance is given by

We use ρ = 0.1. Both I and T (0.1) satisfy conditions (7.4a), (7.4b) and (7.6) (Zhao and Yu, 2007). For = I , θ u = θ ℓ = 1, while for = T (0.1), θ u ≈ 1.84 and θ ℓ ≈ 0.46 (Wainwright, 2006), for the uncompressed lasso in (2.6) and in (2.7).

In the following simulations, we carry out the lasso using procedure lars(Y, X ) that implements the LARS algorithm of Efron et al. (2004) to calculate the full regularization path; the parameter λ is then selected along this path to match the appropriate condition specified by the analysis. For the uncompressed case, we run lars(Y, X ) such that

and for the compressed case we run lars( Y, X ) such that

In each individual plot shown below, the covariance = 1 n E X T X and model β * are fixed across all curves in the plot. For each curve, a compression factor f ∈ {5, 10, 20, 40, 80, 120} is chosen for the compressed lasso, and we show the probability of success for recovering the signs of β * as the number of compressed observations m increases, where m = 2θ σ 2 s log( ps) + s + 1 for θ ∈ [0.1, u], for u ≥ 3. Thus, the number of compressed observations is m, and the number of uncompressed observations is n = f m. Each point on a curve, for a particular θ or m, is an average over 200 trials; for each trial, we randomly draw X n× p , m×n , and ǫ ∈ R n . However β * remains the same for all 200 trials, and is in fact fixed across different sets of experiments for the same sparsity level.

We consider two sparsity regimes:

for α ∈ {0.1, 0.2, 0.4} (6.5a)

Fractional power sparsity: s( p) = αp γ for α = 0.2 and γ = 0.5. (6.5b)

The coefficient vector β * is selected to be a prefix of a fixed vector β ⋆ = (-0.9, -1.7, 1.1, 1.3, 0.9, 2, -1.7, -1.3, -0.9, -1.5, 1.3, -0.9, 1.3, 1.1, 0.9) T (6.6)

That is, if s is the number of nonzero coefficients, then

As an exception, for the case s = 2, we set β * = (0.9, -1.7, 0, . . . , 0) After each trial, lars(Y, X ) outputs a “regularization path,” which is a set of estimated models P m = {β} such that each β ∈ P m is associated with a corresponding regularization parameter λ(β), which is computed as

The coefficient vector β ∈ P m for which λ( β) is closest to the value λ m is then evaluated for sign consistency, where

If sgn( β) = sgn(β * ), the trial is considered a success, otherwise, it is a failure. We allow the constant c that scales λ m to change with the experimental configuration (covariance , compression factor f , dimension p and sparsity s), but c is a fixed constant across all m along the same curve.

Table 1 summarizes the parameter settings that the simulations evaluate. In this table the ratio m/ p is for m evaluated at θ = 1. The plots in Figures 1234show the empirical probability of the event E(sgn( β) = sgn(β * )) for each of these settings, which is a lower bound for that of the event {supp( β) = supp(β * )}. The figures clearly demonstrate that the compressed lasso recovers the true sparsity pattern as well as the uncompressed lasso.

We now study the behavior of predictive and empirical risks under compression. In this section, we refer to lasso2(Y ∼ X, L) as the code that solves the following ℓ 1 -constrained optimization problem directly, based on algorithms described by Osborne et al. (2000):

Let us first define the following ℓ 1 -balls B n and B n,m for a fixed uncompressed sample size n and dimension p n , and a varying compressed sample size m. By Greenshtein and Ritov (2004), given a sequence of sets of estimators

the uncompressed Lasso estimator β n as in (4.7) is persistent over B n . Given n, p n , Theorem 4.1 shows that, given a sequence of sets of estimators

for log 2 (np n ) ≤ m ≤ n, the compressed Lasso estimator β n,m as in (4.24) is persistent over B n,m .

We use simulations to illustrate how close the compressed empirical risk computed through (6.21) is to that of the best compressed predictor β * as in (4.23) for a given set B n,m , the size of which depends on the data dimension n, p n of an uncompressed design matrix X , and the compressed dimension m; we also illustrate how close these two type of risks are to that of the best uncompressed predictor defined in (4.6) for a given set B n for all log np n ≤ m ≤ n.

We let the row vectors of the design matrix be independent identical copies of a random vector X ∼ N (0, ). For simplicity, we generate Y = X T β * + ǫ, where X and β * ∈ R p , E (ǫ) = 0 and E ǫ 2 = σ 2 ; note that E (Y |X ) = X T β * , although the persistence model need not assume this.

Note that for all m ≤ n,

Hence the risk of the model constructed on the compressed data over B n,m is necessarily no smaller than the risk of the model constructed on the uncompressed data over B n , for all m ≤ n.

For n = 9000 and p = 128, we set s( p) = 3 and 9 respectively, following the sublinear sparisty (6.5a) with α = 0.2 and 0.4; correspondingly, two set of coefficients are chosen for β * , β * a = (-0.9, 1.1, 0.687, 0, . . . , 0) T (6.14) so that β * 1 < L n and β * a ∈ B n , and β * b = (-0.9, -1.7, 1.1, 1.3, -0.5, 2, -1.7, -1.3, -0.9, 0, . . . , 0) T (6.15)

In order to find β * that minimizes the predictive risk R(β) = E (Y -X T β) 2 , we first derive the following expression for the risk. With = A T A, a simple calculation shows that

For the next two sets of simulations, we fix n = 9000 and p n = 128. To generate the uncompressed predictive (oracle) risk curve, we let

Hence we obtain β * by running lasso2(

To generate the compressed predictive (oracle) curve, for each m, we let

(6.19)

Hence we obtain β * for each m by running lasso2(

). We then compute oracle risk for both cases as

(6.20)

For each chosen value of m, we compute the corresponding empirical risk, its sample mean and sample standard deviation by averaging over 100 trials. For each trial, we randomly draw X n× p with independent row vectors x i ∼ N (0, T (0.1)), and Y = Xβ * + ǫ. If β is the coefficient vector returned by lasso2( Y ∼ X, L n,m ), then the empirical risk is computed as

where Q n×( p+1) = [Y, X ] and γ = (-1, β 1 , . . . , β p ).

We first state a result which directly follows from the analysis of Theorem 3.4, and we then compare it with the Gaussian ensemble result of Wainwright (2006) that we summarized in Section 2.

First, let us state the following slightly relaxed conditions that are imposed on the design matrix by Wainwright (2006), and also by Zhao and Yu (2007), when X is deterministic:

where min (A) is the smallest eigenvalue of A. In Section 7.B, Proposition 7.4 shows that Sincoherence implies the conditions in equations (7.1a) and (7.1b).

From the proof of Theorem 3.4 it is easy to verify the following. Let X be a deterministic matrix satisfying conditions specified in Theorem 3.4, and let all constants be the same as in Theorem 3.4. Suppose that, before compression, we have noiseless responses Y = Xβ * , and we observe, after compression, X = X , and

where m×n is a Gaussian ensemble with independent entries: i, j ∼ N (0, 1/n), ∀i, j , and ǫ ∼ N (0, σ 2 I m ). Suppose m ≥ 16C 1 s 2 η 2 + 4C 2 s η (ln p + 2 log n + log 2(s + 1)) and λ m → 0 satisfies (3.13). Let β m be an optimal solution to the compressed lasso, given X , Y , ǫ and λ m > 0:

Then the compressed lasso is sparsistent:

that the upper bound on m ≤ n 16 log n in (3.12) is no longer necessary, since we are handling the random vector ǫ with i.i.d entries rather than the non-i.i.d ǫ as in Theorem 3.4.

We first observe that the design matrix X = X as in (7.2) is exactly a Gaussian ensemble that Wainwright (2006) analyzes. Each row of X is chosen as an i.i.d. Gaussian random vector ∼ N (0, ) with covariance matrix = 1 n X T X . In the following, let min ( S S ) be the minimum eigenvalue of S S and max ( ) be the maximum eigenvalue of . By imposing the S-incoherence condition on X n× p , we obtain the following two conditions on the covariance matrix , which are required by Wainwright (2006) for deriving the threshold conditions (2.6) and (2.7), when the design matrix is a Gaussian ensemble like X : When we apply this to X = X where is from the Gaussian ensemble and X is deterministic, this condition requires that

In addition, it is assumed in Wainwright (2006) that there exists a constant C max such that max ( ) ≤ C max .

(7.6) This condition need not hold for 1 n X T X ; In more detail, given max (

we first obtain a loose upper and lower bound for X 2 2 through the Frobenius norm X F of X . Given that X j (7.7) which implies that 1 ≤ max ( 1 n X T X ) ≤ p. Since we allow p to grow with n, (7.6) need not hold. Finally we note that the conditions on λ m in the Gaussian Ensemble result of Wainwright (2006) are (3.13 a) and a slight variation of (3.13 b):

hence if we further assume that ( 1 n X T S X S ) -1 ∞ ≤ D max for some constant D max ≤ +∞, as required by Wainwright (2006) on Hence by imposing the S-incoherence condition on a deterministic X n× p with all columns of X having ℓ 2 -norm n, when m satisfies the lower bound in (3.12), rather than (2.6) with θ u = C max η 2 C min with C max as in (7.6), we have shown that the probability of sparsity recovery through lasso approaches one, given λ m satisfies (3.13), when the design matrix is a Gaussian Ensemble generated through X with m×n having independent i, j ∈ N (0, 1/n), ∀i, j . We do not have a comparable result for the failure of recovery given (2.7).

Proof. Given that A 2 < 1, I 2 = 1, and by Proposition 7.2, min (

Proposition 7.4. The S-Incoherence condition on an n × p matrix X implies conditions (7.1a) and (7.1b).

Proof. It remains to show (7.1a) given Proposition 7.3. Now suppose that the incoherence condition holds for some η ∈ (0, 1], i.e., 1 n X T S c X S ∞ + A ∞ ≤ 1η, we must have

(7.17)

Finally, we have

C. Proof of Lemma 3.5

where g i j , ∀i = 1, . . . , m, j = 1, . . . , n are independent N (0, 1) random variables. We define (7.19) and we thus have the following: (7.20b) where Y ℓ , ∀ℓ, are independent random variables, and

Let us define a set of zero-mean independent random variables Z 1 , . . . , Z m ,

In the following, we analyze the integrability and tail behavior of Z ℓ , ∀ℓ, which is known as “Gaussian chaos” of order 2.

We first simplify notation by defining Y := n k=1 n j =1 g k g j x k y j , where g k , g j are independent N (0, 1) variates, and Z , (7.24) where E (Z ) = 0. Applying a general bound of Ledoux and Talagrand (1991) for Gaussian chaos gives that

for all q > 2.

The following claim is based on (7.25), whose proof appears in Rauhut et al. (2007), which we omit. ∀q > 2, E Z q ≤ q!M q-2 s/2.

Clearly the above claim holds for q = 2, since trivially E |Z | q ≤ q!M q-2 s/2 given that for (7.27e) given that x 2 , y 2 ≤ 1.

Thus for independent random variables Z i , ∀i = 1, . . . , m, we have (7.28) where

≤ 2.5044, ∀i .

Finally, we apply the following theorem, the proof of which follows arguments from Bennett (1962):

Theorem 7.6. (Bennett Inequality (Bennett, 1962)) Let Z 1 , . . . , Z m be independent random variables with zero mean such that (7.29) for every q ≥ 2 and some constant M and v i , ∀i = 1, . . . , m. Then for x > 0,

We can then apply the Bennett Inequality to obtain the following:

≈ 2.5044 and C 2 = √ 8e ≈ 7.6885.

We use Lemma 3.5, except that we now have to consider the change in absolute row sums of 1 n X T S c X S ∞ and A ∞ after multiplication by . We first prove the following claim.

Claim 7.7. Let X be a deterministic matrix that satisfies the incoherence condition. If (7.32) for any two columns X i , X j of X that are involved in (3.21b), then (7.33) and min 1 m Z T S Z S ≥ ηsτ.

(7.34)

Proof. It is straightforward to show (7.33). Since each row in 1 m ( X ) T S c ( X ) S and A has s entries, where each entry changes by at most τ compared to those in 1 n X T X , the absolute sum of any row can change by at most sτ ,

We now prove (7.34). Defining E = A -A, we have (7.37) given that each entry of A deviates from that of A by at most τ . Thus we have that

(7.39d)

We let E represents union of the following events, where τ = η 4s :

1

Consider first the implication of E c , i.e., when none of the events in E happens. We immediately have that (3.21b), (7.34) and (3.22b) all simultaneously hold by Claim 7.7; and (3.21b) implies that the incoherence condition is satisfied for Z = X by Proposition 7.4.

We first bound the probability of a single event counted in E. Consider two column vectors x =

given that τ = η 4s . We can now bound the probability that any such large-deviation event happens. Recall that p is the total number of columns of X and s = |S|; the total number of events in E is less than p(s + 1). Thus

We first show that each of the diagonal entries of T is close to its expected value.

We begin by stating state a deviation bound for the χ 2 n distribution in Lemma 7.8 and its corollary, from which we will eventually derive a bound on

Corollary 7.9. (Deviation Bound for Diagonal Entries of T ) Given a set of independent normally distributed random variables X 1 , . . . , X n ∼ N (0, σ 2 X ), for 0 ≤ ǫ < 1 2 ,

Thus by Lemma 7.8, we obtain the following:

Therefore we have the following by a union bound, for ǫ < 1 2 ,

We next show that the non-diagonal entries of T are close to zero, their expected value.

Lemma 7.10. (Johnstone (2001)) Given independent random variables X 1 , . . . , X n , where X 1 = z 1 z 2 , with z 1 and z 2 being independent N (0, 1) variables, Y n , where Y i = x 1 x 2 is a product of two independent normal random variables x 1 , x 2 ∼ N (0, σ 2 X ), we have

Proof. First, we let Thus we have the following (7.52b) and thus the statement in the Corollary.

We are now ready to put things together. By letting each entry of m×n to be i.i.d. N (0, 1 n ), we have for each diagonal entry D = n i=1 X 2 i , where

where the last inequality is obtained by plugging in ǫ = b log n n 3 and σ 2 X = 1 n in (7.44).

For a non-diagonal entry W = n i=1 Y i , where (7.56) by plugging in σ 2 X = 1 n in ( 7.52a) directly. Finally, we apply a union bound, where b = 2 for non-diagonal entries and b = 16 for diagonal entries in the following:

given that m 2 ≤ n b log n for b = 2.

Recall that Z = X = X , W = Y = Y , and ω = ǫ = ǫ, and we observe W = Zβ * + ω.

First observe that the KKT conditions imply that β ∈ R p is optimal, i.e., β ∈ m for m as defined in (3.5), if and only if there exists a subgradient

which is equivalent to the following linear system by substituting W = Zβ * + ω and re-arranging,

→ 0, (7.72c) by (7.70) and the fact that by (3.10), 1 + A ∞ ≤ 2η.

We first prove the following.

Claim 7.12. If m satisfies (3.12), then 1 m max i, j (B i, j ) ≤ 1 + η 4s .

Proof. Let us denote the i th column in Z S with Z S,i . Let x = Z S,i and y = Z S, j be m × 1 vectors. By Proposition 3.6, x 2 2 , y 2 2 ≤ m (1 + η 4s . We have by function of x, y, Remark 7.13. In fact, max i, j M i, j = max i,i M i,i .

The results presented here suggest several directions for future work. Most immediately, our current sparsity analysis holds for compression using random linear transformations. However, compression with a random affine mapping X → X + may have stronger privacy properties; we expect that our sparsity results can be extended to this case. While we have studied data compression by random projection of columns of X to low dimensions, one also would like to consider projection of the rows, reducing p to a smaller number of effective variables. However, simulations suggest that the strong sparsity recovery properties of ℓ 1 regularization are not preserved under projection of the rows.

It would be natural to investigate the effectiveness of other statistical learning techniques under compression of the data. For instance, logistic regression with ℓ 1 -regularization has recently been shown to be effective in isolating relevant variables in high dimensional classification problems (Wainwright et al., 2007); we expect that compressed logistic regression can be shown to have similar theoretical guarantees to those shown in the current paper. It would also be interesting to extend this methodology to nonparametric methods. As one possibility, the rodeo is an approach to sparse nonparametric regression that is based on thresholding derivatives of an estimator (Lafferty and Wasserman, 2007). Since the rodeo is based on kernel evaluations, and Euclidean distances are approximately preserved under random projection, this nonparametric procedure may still be effective under compression.

The formulation of privacy in Section 5 is, arguably, weaker than the cryptographic-style guarantees sought through, for example, differential privacy (Dwork, 2006). In particular, our analysis in terms of average mutual information may not preclude the recovery of detailed data about a small number of individuals. For instance, suppose that a column X j of X is very sparse, with all but a few entries zero. Then the results of compressed sensing (Candès et al., 2006) imply that, given knowledge of the compression matrix , this column can be approximately recovered by solving the compressed sensing linear program min X j 1 (8.1a) such that Z j = X j . (8.1b) However, crucially, this requires knowledge of the compression matrix ; our privacy protocol requires that this matrix is not known to the receiver. Moreover, this requires that the column is sparse; such a column cannot have a large impact on the predictive accuracy of the regression estimate. If a sparse column is removed, the resulting predictions should be nearly as accurate as those from an estimator constructed with the full data. We leave the analysis of this case this as an interesting direction for future work.

T .

T

This terminology is due to Pradeep Ravikumar.