A Path Algorithm for Constrained Estimation

Many least squares problems involve affine equality and inequality constraints. Although there are variety of methods for solving such problems, most statisticians find constrained estimation challenging. The current paper proposes a new path following algorithm for quadratic programming based on exact penalization. Similar penalties arise in $l_1$ regularization in model selection. Classical penalty methods solve a sequence of unconstrained problems that put greater and greater stress on meeting the constraints. In the limit as the penalty constant tends to $\infty$, one recovers the constrained solution. In the exact penalty method, squared penalties are replaced by absolute value penalties, and the solution is recovered for a finite value of the penalty constant. The exact path following method starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, slides along, and exits from the various constraints. Path following in lasso penalized regression, in contrast, starts with a large value of the penalty constant and works its way downward. In both settings, inspection of the entire solution path is revealing. Just as with the lasso and generalized lasso, it is possible to plot the effective degrees of freedom along the solution path. For a strictly convex quadratic program, the exact penalty algorithm can be framed entirely in terms of the sweep operator of regression analysis. A few well chosen examples illustrate the mechanics and potential of path following.

💡 Research Summary

The paper introduces a novel path‑following algorithm for solving quadratic programming problems that arise in constrained least‑squares estimation. The authors consider the minimization of a strictly convex quadratic function
(f(x)=\tfrac12 x^{\top}Ax+b^{\top}x+c)
subject to affine equality constraints (Vx=d) and affine inequality constraints (Wx\le e). Classical penalty methods replace the constraints with squared penalties and require the penalty parameter to tend to infinity in order to recover the constrained solution. In contrast, the exact‑penalty approach substitutes absolute‑value penalties for the constraints, yielding a penalized objective
(E_{\rho}(x)=f(x)+\rho\sum_i|v_i^{\top}x-d_i|+\rho\sum_j\max{0,w_j^{\top}x-e_j}).
When (\rho) exceeds the magnitude of all Lagrange multipliers at the true constrained optimum, minimizing (E_{\rho}) is equivalent to solving the original constrained problem.

The algorithm starts at (\rho=0) where the solution is the unconstrained minimizer (x(0)=-A^{-1}b). As (\rho) increases, the solution (x(\rho)) and the sub‑gradient coefficients (s_i) (for equalities) and (t_j) (for inequalities) evolve linearly until a “event” occurs: either an inactive constraint becomes active (a hitting time) or an active constraint’s coefficient reaches a boundary of its sub‑differential (an escape time). The authors derive explicit formulas for these event times: \