On the Optimal Solution of Weighted Nuclear Norm Minimization

On the Optimal Solution of Weighted Nuclear Norm Minimization Qi Xie a , Deyu Meng a , Shuhang Gu b , L ei Zhang b , Wangmeng Zuo c , Xiangchu Feng d and Zongben Xu a a School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China b Dept. of Computing, The Hong Kong Polytechnic Universit y, Hong Kong c Dept. of Computer Science, Harbin I nstitute of Technolog y , Harbin, China d Dept. of Applied Mathematics, Xidian University , Xi’an, China Abstract In recent years, the nuclear nor m minimization (NNM ) problem has been attrac ting much attention in computer vision and machine lear ning. T he NNM proble m is capitalized on its co nvexity and it can be solved efficiently . The standar d nuclear norm regularizes all singular values eq ually, which is ho wever not flexible enough to fit real scenarios . Weighted nuclear nor m minimization (WNNM) is a natural extension and generalization o f NNM. By assigning p roperly different weights to di fferent sing ular values , WNNM can lead to state- of - the-art results in app lications such as ima ge denoising [3] . Never theless, so far th e global opti mal solution o f WNNM proble m is not co mpletely solved yet d ue to its non -convexity in general ca ses. In thi s article, we stud y the theoretical prop erties of WNNM and prove that WNNM ca n be equivalent ly transfor med into a quadratic pr ogramming pro blem with linear constr aints. This implies that W NNM is equi valent to a convex prob lem and its global optimum can be readily ac hieved by off -the-she lf con vex opti mization solvers. We further show that when the weights are non- de scendin g, the glo bally opti mal solution of WNNM ca n be obtained in closed- form. 1. Intr oduction As a classica l technique for low rank matrix approximation , recently the nuclear n orm minimization (NNM) [1-2 ] has been attracting much attention in co mputer vision a nd machine learning research . T he standard nuclear norm of a matrix     is defined as the sum o f all its si ngular values, i.e.,          󰇛  󰇜  , where   󰇛  󰇜 is the i - th singular value o f  . Nuclear norm is the tightest convex relaxation of t he rank penalt y o f a matrix . Let     be the given data matrix. T he standard NNM problem ai ms to fi nd an ap proximation matrix  o f  by minimizing t he following energy function:                     (1) where  is a positive regularization parameter . It has been shown that the above NNM problem has a closed-form solution [ 2]:      󰇛  󰇜    wher e      is the SVD of  , and   󰇛  󰇜   󰇛      󰇜 . Al beit easy to solve, the NNM model ha s so me limitations. The nuclear norm trea ts a ll the si ngular values equally , and it ignores the p rior knowledge we often have on the matrix singular v alues. For exam ple, in many vision ap plications, the larger singular values o f the data matrix are usually more i mportant than the s maller ones since the y represent the main co mponents of the data . Intuitively , we should assign different weights to different singular values to ma ke NNM more flexible to fit r eal scenarios. T o improve the f lexibility of NNM , researchers have prop osed the weighted nuclear norm m i nimization (WNNM) [3 -4] problem . The weighted unclear norm of a matrix  is defined as:            󰇛  󰇜      (2) where   󰇛  󰇜    󰇛  󰇜      󰇛  󰇜 ,   󰇟           󰇠 and     is the weight assigned to   󰇛  󰇜 . W eig hted nu clear nor m is not a real norm since it does not al ways satisfy the trian gle inequality . The W NNM problem is then for mulated as [3] :                    (3) The WNNM prob lem, however, is m or e dif ficult to solve t han t he NNM problem due to its non-con vexity i n general cases of weights. Gu et al. [3] and Y ang et al. [4] have i ndependently d iscussed t he solution of W NNM and successfully applied WNNM to image denoising and 3D reconstruction. Nonetheless , the globally optimal solution of WNNM is not co mpletely solved yet. I n this article, we aim to study and present t he optimal so lution of WNNM in Eq. (3). W e prove that the WNNM problem can be eq uivalently transfor med into a co nvex pro gramming proble m , and its global ly o ptimal solution can be readily achieved by e mploying o ff-the-shelf quadratic programming techniq ues . W e further show tha t in a sp ecific but very useful case, i.e. , the weights are in a non-desce nding o rder, the global m ini mum can be analytically o btained in close d form. 2. Low-Rank Minimization with W eighted Nuclear Norm W e first give the following Lemma 1 [3] , which builds the important relatio nship between the nu clear norm and the trace of a matrix. Lemma 1[3]. For any     and a diagonal non-neg ative matrix     with non-ascending or d er ed diagonal elemen ts, let      be the SVD o f  , we have    󰇛  󰇜   󰇛  󰇜          󰇟    󰇠  wher e  is the id entity matrix,   󰇛󰇜 and   󰇛  󰇜 ar e the i-th singular va lues of matrices  and  , r espectively . When     and     󰇟    󰇠 r ea ches its maximum value. Based o n the result o f Lemma 1, w e can ha ve the following theore m, which implies an equivalent conve x transform of the original non- convex WNNM problem in E q. (3). Theorem 1. For a ny     , without loss of generality we assume that    , and let      be the SVD of  , wher e   󰇡  󰇛          󰇜  󰇢 . The solution of the WNNM pr o blem in E q. (3) can be expressed as       , wher e   󰇡  󰇛           󰇜  󰇢 is a diagona l non -negative matrix and 󰇛          󰇜 is the solution o f the following convex optimization pr o blem:           󰇛      󰇜                          (4) Proof . For any     , its SVD can b e expressed as        , where   and   are unitar y matric es , and   󰇡  󰇛           󰇜  󰇢 with              Then we have                                                                               󰇛      󰇜                                                           󰇛      󰇜                  According to Lemma 1, we have               󰇛      󰇜         and the optimal solutio n is obtained at     and      . W e then have                                                                                                 󰇛      󰇜                          Fr om the above derivatio n, we can see that the op timal solution o f the WNNM problem in Eq. (3) is       , (5) where D  is the opti mum of the constrained o ptimization prob lem in Eq. (4) . The proof is then co mpleted .  Theorem 1 shows that the WNNM problem can be equivalently trans formed into t he proble m in E q. ( 4). I t is interesting to see t hat Eq. (4) is a conv ex p roblem. T his means t hat the o riginal dif ficul t non-con vex proble m i s equivalent to a convex problem which is much easier to solve. Further more, in a specific yet very useful case , i.e., the weights     are in a non -descending order , it can be sh own that the global optimum of Eq. (4) has a closed for m . We have the follo wing corollary. Corollary 1 . If the weig hts satisfy              , the global ly optimal solution of Eq. (4) is                      , wher e      󰇛       󰇜 . Proof . I f w e ignore the constraint s of the pr oblem in Eq . (4), we can ob tain the f ollo wing unconstrained problem:      󰇛      󰇜           (6) Since     ,       , the above problem is equivalent to the followin g problem:        󰇡      󰇢  . Then it is easy to see that the global ly optimal solu tion of the problem in Eq. (6 ) is       󰇡        󰇢         Since                       , we have                T hus ,     satisfy the constraint of Eq. (4), and this implies that t hey are the solutio n of the original constrained proble m in Eq. (4 ). The proof is then completed.  The conclusion in Corollary 1 is very u seful i n so me real scenarios . For instance, Gu et al. [3] ha ve shown that b y assi gning s maller weights to the larger singular val u es , WNNM lead s to state- of -the-art i mage d enoising results. Actually, co mbining T heorem 1 and C orollar y 1, we can read ily get the globally optimal solution   in the following analytical form (whe n              ):         (7) where                      ,     󰇛       󰇜 , (8) and      is SVD of  . This reveals the un derlying reason of the effectiveness of the W NNM denoising method in [3] . Finally , we s ummarize ho w to calculate the global o ptimum of the WNNM problem as follo ws:  I f the weights sati sfy              , the global ly op timal solutio n of the WNNM pr oblem with input matrix  ca n be e xp r essed as        ， wher e    is the SVD of  and                      ha s a closed form solutio n as pr esented in Co r ollary 1.  I n the general case, i.e., the weights are in an arbitrary order , the glob al op timum of the WNNM problem with input matrix  can be expressed as       ， wher e    is the SVD of  and                     can be calculated b y quadratic programming with linear constraints:               󰇛     󰇜                                  It sho uld be noted th at the above quadratic pr ogramming pr ob lem ca n be a ccurately an d efficien tly solved by ma ny off-the- shelf optimization toolkits [5, 6, 7] . 3. Conclusion In this article , we studied the theoretical properties of the WNNM problem . When the weights ar e in a n arbitrary order , we sho w ed that W NNM can be equivalentl y tran sformed i nto a quadratic p rogramming pro blem which is easy to solve by off-the-shelf toolkits. When the weights are i n a non-descending order, interestingly, we proved that t he globally opti mal solutio n of W NNM can b e o btained in closed form . Our findings re veal that although the W NNM problem is non-convex , its global optimum can sti ll be obtained . It is thus expect ed that the WNNM model will have more success ful applicatio ns in computer visio n and machine learning . Refer enc es [1] E. J. Cand è s and B . Rec ht. Exact m atrix co mpletion via convex optimizatio n. Foundations of Computational Mathe matics, 9(6 ):717 – 772, 2009 . [2] J. - F. Cai, E. J. Candè s, and Z. She n. A singular value threshold ing al gorithm for matrix c ompletion. SIAM Journal on Opti mization, 20(4):1 956 – 1 982, 2010. [3] S. Gu, L . Zhang, W. Zuo, and X. Feng. Weighted Nuclear Norm Minimization with Application to Image Denoising . In CVPR 201 4. [4] L . Y ang, Y . Lin, Z. Lin, T . Lin, and Hon gbin Zha, Facto rization for P rojective and Metric Reconstructio n via W eighted Nuclear Norm Minimizatio n, to appear in Neuro computing. [5] Frank M, W olfe P . An al gorithm for quadratic programming . Naval research lo gistics q uarterly , 3(1-2): 95 -1 10 , 1956. [6] Murty K G. Linear complementarity , linea r and nonlinear p rogramming. Berlin: He ldermann, 198 8. [7] Delbos F , Gi lbert J C. Global linear co nvergence of an a ugmented La grangian al gorithm for solving convex qu adratic optimization prob lems. 2003.