Polynomial time algorithms for bi-criteria, multi-objective and ratio problems in clustering and imaging. Part I: Normalized cut and ratio regions

P olynomia l time alg orithms fo r bi-criteria, multi-ob jectiv e and ratio pro blems in clustering and imaging P art I: Normalized cut and ra tio regi ons Dorit S. Ho c hbaum email: hochbaum@ieor.berk eley.edu F eb 10, 2008 Abstract Partitioning and grouping of similar ob jects plays a fundamental role in image s e gmen- tation and in clustering pro blems. In suc h problems a t ypical goal is t o gro up together similar ob jects, or pixels in the case o f ima ge pro cess ing. A t the s a me time ano ther goal is to have each gr oup distinctly dissimilar from the rest and pos sibly to hav e the group size fairly larg e . Thes e goals a re o ften co mbined as a ratio optimization pr oblem. One ex ample of suc h problem is the normalized cut proble m, a nother is the ratio regions problem. W e devise here the first polynomial time algorithms solving thes e pro blems optimally . The alg o- rithms are efficient and combinatorial. This contrasts with the heuristic a pproaches used in the ima ge segmentation liter ature that for mulate those problems as nonlinea r o ptimization problems, which are then relaxed and solved with spe c tral techniques in real n umbers . These approaches no t only fail to deliver an optimal solution, but they are als o computatio na lly exp ensive. The algorithms prese nted here use as a subroutine a minimum s, t -cut pro cedure on a r e lated gr aph which is of p olynomial size. The o utput consis ts of the optimal solution to the res p ec tive ratio problem, a s well as a sequence of nested solutio n with resp ect to any relative weight ing of the ob jectiv es of the num e r ator and denominator. An extension of the results her e to bi-criteria and multi-criteria ob jectiv e functions is presented in par t II. 1 In tro d uction The leading c hallenge in the field of imaging is vision grouping, or segmen tation. Grouping is to r ecognize and delineate, automatically , the salie nt ob jects in an imag e. Image segmen tation is equiv alent to partitioning the set of p ixels forming the image, or to clustering its p ixels. High qualit y clustering is often defin ed by multi p le attribu tes. As an optimization prob lem this requires attaining more than one ob jectiv e. The motiv atio n for studying the normalized cut problem is an example of setting an optimization criterion in order to attain tw o goals. One goal is to h av e the selected group’s pixels to b e as dissimilar to th e remainder of the image as p ossible, and the second is to maximize the similarity of the pixels within the group. These t wo ob jectiv es are p resen ted as a m inimization of the r atio of th e fir st to the second. A grea t deal of the literat u re is concerned only with the b ipartitioning of the image . Th at is, the separation of one ob ject segmen t fr om the r est of the image - the bac kground. Even this 1 mo dest goal presents a num b er of computational d ifficulties. Wh ile present in g an image as a graph and the similarit y b et ween pairs of ob jects as a weigh t of an edge, a sim p le minimum 2-cut problem will ac hiev e a partition that minimizes similarit y b et w een the t wo parts. An adve r se phenomenon asso cia ted with the minim um 2-cut optimal solution, th at w as noted h o we ver b y Shi and Malik, [19], an d others, is that often the selected part tends to b e ve r y s m all. T o comp ensate an d correct for the p henomenon of small segmen ts Shi and Malik introdu ced the notion of normalize d cut . Graph theoretical framew ork is suitable for represen ting imag e segmen tation and grouping problems. The image segmen tation p roblem is pr esen ted on an u ndirected graph G = ( V , E ), where V is the set of pixels and E are the p airs of wh ic h s imilarit y information is av ailable. T yp ically one considers a planar image with pixels arran ged along a grid. The 4-neigh b ors set up is commonly u sed with eac h p ixel having 4 neigh b ors t wo along the vertica l axi s and tw o along the horizont al axis. T his set up forms a p lanar grid graph. T he 8-neigh b ors arrangement is also used, b u t the planar structure is n o longer preserved, and complexit y of the v arious heu r istic algorithms is increasing, sometimes significan tly . Images can of course b e also 3-dimensional, and in general clustering problems there is n o grid str ucture. Th e algorithms p resent ed h er e do not assu me an y sp ecific pr op erty of the graph G - they work for general graphs. The edges in the graph representi n g the image carry similarity weigh ts. There is a great deal of literature on how to generate sim ilarity weigh ts, and w e do not discuss this issue here. W e only use th e fact that similarit y is inv ersely increasing w ith the difference in attribu tes b et ween th e p ixels. In terms of the graph, eac h edge [ i, j ] is assigned a sim ilarity w eight w ij that is in cr easing as the tw o pixels i and j are p ercei ved to b e more similar. Lo w v alues of w ij are in terpreted as dissimilarit y . Ho wev er, in some con texts one might wan t to generate dissimilarity w eigh ts indep end ently . In that case eac h edge has t wo w eights, w ij for similarity , and ˆ w ij for dissimilarit y . Tw o ap p lications of efficien t al gorithms for ratio problems are presented: On e for the pr ob - lem of “normalize d c ut” , whic h is to minimize the ratio of the s imilarit y b etw een the set of ob jects and its complement and the similarity within the set of ob jects. The second problem is that of “r atio - r e g ions” which is to min imize the r atio of of the s imilarit y b etw een th e set of ob jects and its complement and the num b er (or weig ht) of the ob jects within the set. T he al- gorithms not only pro vide an optimal solution to the ratio problem, but also delive r a sequence of solutions for all p ossib le relat ive w eighti n g of the t wo ob jectiv es. T h ese solutions are often more inform ativ e than the optimal solution to the ratio problem alone. Although the multi -segmenta tion p roblem is a p artition to m u ltiple sets, th e r atio p roblems discussed here do not d irectly add r essed as suc h, du e to computational iss u es. In stead these bi- partitions hav e b een used recursive ly to generate an y desired n umber of segmen ts. It is sho w n in part I I that all the problems presente d h ere ha ve optimal solutions that are b ipartition, and ho w to c haracterize ratio p roblems in general with this pr op erty . F o r multiple s egments, the Mark o v Random Fields is one mo d el th at has b ee n p opu lar in that con text. It has b een stu d ied b y numerous authors, e.g. [2 ], and has b een established for the firs t time to b e p olynomia l time solv able f or conv ex ob jectiv es in [15]. 2 2 Notation Let the we ights of the edges in th e graph b e w ij for [ i, j ] ∈ E . If the edges ha ve tw o sets of w eight s, these w ill b e denoted by w 1 ij and w 2 ij . A b ipartition of the graph is called a cut , ( S, ¯ S ) = { [ i, j ] | i ∈ S, j ∈ ¯ S } , w h ere ¯ S = V \ S . W e d efine the c ap acity of a cut ( S, ¯ S ), and the capacit y f or any p air of sets ( A, B ) to b e C ( A, B ) = P i ∈ A,j ∈ B w ij . W e defi ne the c ap acity of a set A ⊂ V to b e C ( A ) = C ( A, A ) = P i,j ∈ A w ij . F or inputs with tw o sets of edge weigh ts we let C 1 ( A, B ) = P i ∈ A,j ∈ B w 1 ij and C 2 ( A, B ) = P i ∈ A,j ∈ B w 2 ij . Giv en a partition of a graph into k d isjoin t comp onen ts, { V 1 , . . . , V k } the k-cut v alue is C ( V 1 , . . . , V k ) = 1 2 P k i =1 C ( V i , ¯ V i ). The pr oblem of partitioning a graph to k nonempt y comp o- nen ts that min imize the k -cut v alue is p olynomial time solv able for fi xed k , [12]. F or graphs with weigh ted no des, we let the w eigh t of no de j b e v j . The w eight of a set of no des A is denoted b y V ( A ) = P j ∈ A v j . 3 Sev eral ratio problems W e list h ere four types of ratio p roblems. This include, in addition to the normalized cut problem and the ratio r egions problem, also the densest sub grap h problem and the “rat io cut” problem. W e solv e here only the firs t t wo . Th e third problem has b een known to b e p olynomial time solv able, and the last problem is NP-hard. 3.1 The normalized cut problem Shi and Malik n oted in their work on s egmen tation that cut pro cedures tend to create segmen ts that may b e very small in size. T o address this issue they p r op osed seve r al v ersions of ob jectiv e functions that pro vide larger segmen ts in an optimal solution. Among the prop osed ob jectiv e they formulated the normalized cut as the optimization prob lem min S ⊂ V C ( S, ¯ S ) · ( 1 | S | + 1 | ¯ S | ) . This problem is equiv alent to finding the expander ratio of th e graph discu ssed in the next subsection. This ob jectiv e fu nction drives the segmen t S and its complement to b e approximate ly of equal size. I n deed, lik e the b alanced cut problem the problem w as sho w n to b e NP-hard, [19], b y reduction from set p artitioning. A v arian t of the p roblem also defi ned by Shi and Malik is min S ⊂ V C ( S, ¯ S ) · ( 1 C ( S, V ) + 1 C ( ¯ S , V ) ) . Another v arian t yet of the p roblem is the quant ity h G = min C ( S,V \ S ) min { C ( S,S ) ,C ( V \ S ) } , also kno wn as the Che e ger c onstant , [5, 6]. More frequent ly , for minor v arian ts of the problem, th e de- nominator is min {| S | , | V \ S |} , or | S | is rep laced by a quan tity represent in g the v olume of S . This Cheeger constant is appro ximated b y the second largest eigen v alue of a certain related adjacency matrix of the graph. This eigen v alue λ 1 is r elated to the Ch eeger constant by the inequalities: 2 h G ≥ λ 1 ≥ h 2 G / 2. Computing the v alue of the Cheeger constant is NP-hard - 3 it is the s ame as fin d ing the expander r atio of a graph and again it d riv es to a r oughly equal or balanced partition of the graph. T he dominant tec hniqu es in vision grouping are sp ectral in nature. Th at is, they compute the eigen v alues and the eigen ve ctors an d then some t yp e of rounding pro cess, see e.g. [21, 20]. Instead of the sum problem, there are other related optimization problems used for image segmen tation. Sharon et al. [20] defin e the n ormalized cut as min S ⊂ V C ( S, ¯ S ) C ( S, S ) . Sharon et al. [20] state that: A salient segmen t in the image is one for which the similarit y across its b order is sm all, whereas the similarit y within the segmen t is large (for a mathematical description, see Methods). W e can thus seek a segmen t that min imizes the ratio of these t wo expressions. Despite its conceptual us efu lness, m in imizing this n orm alized cut measure is computationally prohibitiv e, with cost that incr eases exp onen tially with image size . One of our con trib utions here is to sh o w that the p roblem of minimizing this ratio is in f act solv able in p olynomial time, and with a com b inatorial algo rith m . The t ypical solution approac h us ed when addr essin g optimiza tion problems for image seg- men tation is to app ro ximate th e p r oblem ob jectiv e by a nonlinear (quadr atic) expression f or whic h the eig env ectors of an asso ciated matrix form an optimal s olution. Let binary v ariables x i for i ∈ V b e defined so that x i = 1 if no de i in th e selected side of the cut – the segment . The f ollo wing nonlinear form ulation is th e relaxa tion that has b een used by Sharon et al. an d others, [20, 21, 19]: min P w ij ( x i − x j ) 2 P w ij x i · x j = x T L x x T W x , where L is the Laplacian matrix of the graph and W is a matrix appropr iately defined. T h e use of sp ectral tec hniques inv olv es real n umb er compu tations with the associated numerical issues. Even an exact solution to the nonlinear problem is a vec tor of r eal num b ers wh ereas the original p r oblem is discrete and b inary . Ho w ever, this n ormalized cut problem (without the “balanced” requiremen t) is p olynomial time solv able. W e sho w an algorithm solving the p roblem in the same complexit y as a single minim u m s, t -cut on a r elated graph on O ( n + m ) no d es and O ( n + m ) edges. 3.2 Ratio regions and expanders Consider the ob jectiv e function min | S |≤ n 2 C ( S, ¯ S ) | S | . Th is v alue of the optimal solution, for a graph G , is kno wn as the expansion r atio of G . This problem is NP-hard as the limit on the size of | S | mak es it d ifficult, and d r iv es the solution to wards a b alanc e d cut – a kno wn NP-h ard problem. The ob jectiv e function can also be written as min S ⊂ V C ( S, ¯ S ) min {| S | , | ¯ S | } . A v arian t of this problem min S ⊂ V C ( S, ¯ S ) | S | has b ee n consid ered und er the name r atio r e gions b y Co x et al. [7]. The ratio region problem is m otiv ated by seeking a segmen t, or region, w h ere 4 the b ou n dary is of lo w cost and the segment itself has high wei ght. Th e p roblem stud ied b y Co x et al. is restricted to p lanar graph s and th us to planar grid images w ith 4 neigh b ors only . Though the b oundary of the region is defined as a path, the path corresp onds to a cut in the dual graph. F or graph n o des of weigh t d i the problem is min S ⊂ V C ( S, ¯ S ) P i ∈ S d i . This problem is shown here to b e p olynomially solv able b y a parametric cut pro cedure, in the complexit y of a single minimum cut. The problem is in fact equiv alen t to a binary and linear version of the Mark o v Random Fields pr oblem, called the maxim u m s -excess problem in [14]. I t is in teresting to note that the pseud oflo w algorithm in [14] is set to solve the maximum s -excess problem d irectly . Our algorithm for the ratio regions p roblem applies for no de wei ghts that can b e either p ositiv e or n egativ e. Th is generalizes the application con text of Cox et al. the no de we ighs w ere all p ositiv e. 3.3 Densest subgraph Sark ar and Bo yer [18] d efined th e p r oblem min S ⊂ V C ( S,S ) | S | . This ob jectiv e is of inte r est for w eight s that reflect d iss imilarit y . In that case the goal is to minimize the dissimilarit y w ith in the selected segmen t while the size of the segmen t te n ds to b e large. F or sim ilarity wei ghts the ob jectiv e wo u ld b e to maximize this quantit y . Both this problem, and its maximization v ersion are solve d in p olynomial time. Also the no des can carry arbitrary weigh ts and the algorithm is still applicable with no c han ge in th e ru nning time. This problem has b een kno w n for a long time as the maxim um d ensit y sub graph is the subgraph ind uced b y the subset of n o des D maximizing, max S ⊂ V C ( S, S ) | S | . This problem was sh own to b e solv able in p olynomial time by Goldb erg [10]. Gallo, Grigori- adis and T arjan [9] show ed ho w the p roblem w ould b e solv ed as a parametric minimum s , t -cut in the co m plexit y of a single s, t -cut. A no de w eight ed v ersion of the pr oblem is max S ⊂ V C ( S,S ) V ( S ) . This problem is solve d by a minor extension of the den sest subgraph app roac h in th e same ru n time. 3.4 “Ratio cuts” This pr ob lem was in tro duced b y W ang an d Siskind [22]. In the ratio cut problem the edges ha ve t wo weigh ts asso ciated with eac h. W ang and Siskind studied the case where w 1 ij are p ositive and w 2 ij are equal to 1 for all [ i, j ] ∈ E . The goal is to m inimize the r atio, min S ⊂ V C 1 ( S, ¯ S ) C 2 ( S, ¯ S ) . This pr oblem wa s shown in [22] to b e at least as hard as th e sp ars est cut p r oblem, and therefore NP-hard. On the other h an d , for planar graphs, W ang and S iskind d emonstrated that the problem is solv able in p olynomial time. 5 3.5 Ov erview The problems, that the metho d ology paradigm pr esen ted here solve in p olynomial time, are summarized in the T able 1: Problem nam e Ob jectiv e Reference Normalized cut min S ⊂ V C ( S, ¯ S ) C ( S,S ) [19, 20] Normalized cut’ min C ( S, ¯ S ) C ( S,V ) [19] “Densit y ” min C ( S,S ) | S | [18] Ratio regions min C ( S, ¯ S ) | S | [7] W eighted ratio regions min C ( S, ¯ S ) P i ∈ S d i [7] T able 1: Ratio optimiza tion problems in image segmen tation W e presen t her e in detail only the algo r ithm for the normalized cut problem, wh ich is the hardest one on the list. The constr u ction for the ratio regions is describ ed briefly . F or the “densit y” problem with either a min imization or maximization ob jectiv e fu nction we studied the pr oblem for the en tire sequen ce of solutio n s in the con text of dynamically ev olving fac ility set, see [17]. Thus for the ”density” case we only p oin t out an efficien t algo r ithm. 4 The solution approac h 4.1 Monotone in teger progr amming form ulation The k ey is to formulate the problem as an int eger linear pr ogramming problem, a 0-1 in teger programming here, with monotone inequalitie s constraint s. It was sh own in [16] that an y in teger p rogramming formulatio n on monotone constrain ts has a corresp ond ing graph where the min im um cu t solution corresp onds to the optimal solution to the in teger programming problem. Thus the form ulation is solv able in p olynomial time. T o con vert the ratio ob jectiv e to a linear ob jectiv e we utilize the redu ction of the ratio problem to a linearized optimiza tion problem. 4.2 Linearizing ratio problems A general approac h for maximizing a fractional (or as it is s ometimes call ed, geometric) ob jec- tiv e fun ction o v er a feasible region F , min x ∈F f ( x ) g ( x ) , is to reduce it to a sequence of calls to an oracle that pro vides the y es/no answer to the λ - qu estion : Is there a feasible subset V ′ ⊂ V such that P x ∈F f ( x ) − λ P x ∈F g ( x ) < 0? If the an s w er to the λ -question is yes then the optimal solution has v alue smaller than λ . Otherwise, th e optimal v alue is greater than or equal to λ . A standard app roac h is then to utilize a binary searc h p ro cedure th at calls for the λ -qu estion O (log ( U F )) times in order to s olve the pr oblem, where U = P [ i,j ] ∈ E w ij , and F = P [ i,j ] ∈ E w ′ ij for the w eights at the den ominator w ′ ij . 6 Therefore, if the linearized v ers ion of the problem, that is the λ -question, is s olved in p olynomial time, then so is the ratio problem. Note that the num b er of calls to the linear optimization is not strongly p olynomial b ut rather, if b inary search is emplo ye d , dep ends on the logarithm of the magnitude of the num b ers in the in put. In some cases how ev er there is an efficien t pro cedure that uses the solution for one parameter v alue to compute the v alue for another p arameter v alue more efficien tly . Such is the case for the densest subgraph p roblem whic h has an efficie nt p arametric pro cedure ([9]). It is imp ortan t to note that not al l ratio p roblems are solv able in p olynomial time. On e prominent example of su ch ratio problem is the r atio cut introdu ced by W ang and Siskind, [22]. That criterion applies in a graph with t w o sets of w eights for eac h edge. Let w 1 ij , w 2 ij b e the t wo weigh ts assigned to edge [ i, j ]. Then a cut with resp ect to w 1 ij separating S from ¯ S is C 1 ( S, ¯ S ) and with r esp ect to w 2 ij is is d enoted by C 2 ( S, ¯ S ). The r atio criterion defined b y W ang and S iskind is to m inimize C 1 ( S, ¯ S ) C 2 ( S, ¯ S ) . F or the weig ht w 2 ij they use th e v alue 1, so this ob jectiv e is to find a cut minimizing the cut v alue divided b y the n umb er of edges in the cut. As in other cases, the rationale is to try and increase the n u m b er of edges in the cut, and hence the size of the cluster/segmen t. Th is particular criterion w as sho wn in [22] to b e NP- hard, and p olynomial time s olv able on planar graph s. F or the ratio cu t problem the linea r ized problem is NP-h ard , and the ratio cut problem is NP-hard as w ell. The lin earized p roblem is NP-h ard b y r eduction from maxim um cut, and the r atio problem b y reduction from the sparsest cut pr oblem, [22]. Ho wev er th e ratio cut problem has a p olynomia l time algorithm for p lanar graphs. F or planar graphs the λ -question is solve d b y fin ding a maxim u m weig ht non-bipartite matc hing in a related graph. The pr o cedure of [22] in d eed mak es rep eat ed calls to s olving n on-bipartite matc hing p roblems, where for eac h v alue of λ another graph has to b e constructed. It is also im p ortan t to note that linearizing is n ot alw a ys the righ t approac h to use for a ratio problem. F or example, the pr oblem of finding a p artition of a graph to k comp onents minimizing the k -cut b et ween components for k ≥ 2 divided by the num b er of components k , alw a ys has an optimal solution with k = 2 wh ic h is attained by a min im um 2-cut algorithm. On the other hand, the linearized problem is muc h harder to solv e (though it can b e solv ed in p olynomial time.) Ad ditional details are pro vid ed in p art I I of this p ap er. 5 The normalized cut form ulation W e fi rst pr o vide a formulatio n f or th e p roblem, min S ⊂ V C 1 ( S, ¯ S ) C 2 ( S,S ) . W e use different similarit y w eight s for the numerator w ′ ij and denominator w ij . W e b egin with an integ er programming f ormulation of the problem. Let x i = ( 1 if i ∈ S 0 if i ∈ ¯ S . W e define additional bin ary v ariables: z ij = 1 if exactl y one of i or j is in S ; y ij = 1 if b oth i or j are in S . z ij = ( 1 if i ∈ S , j ∈ ¯ S , or i ∈ ¯ S , j ∈ S 0 if i, j ∈ S or i, j ∈ ¯ S . 7 y ij = ( 1 if i, j ∈ S or i, j ∈ ¯ S 0 otherwise. With these v ariables the follo w in g is a v alid formulatio n (NC) of the normalized cu t prob lem: (NC) min P w ij z ij P w ′ ij y ij sub ject to x i − x j ≤ z ij for all [ i, j ] ∈ E x j − x i ≤ z j i for all [ i, j ] ∈ E y ij ≤ x i for all [ i, j ] ∈ E y ij ≤ x j 1 ≤ P [ i,j ] ∈ E y ij ≤ | E | − 1 x j binary j ∈ V z ij , y ij binary j ∈ V . T o verify the v alidit y of th e form u lation notice that the ob jectiv e fun ction drive s the v alues of z ij to b e as small as p ossible, and the v alues of y ij to b e as large as p ossible. With the constrain ts, z ij cannot b e 0 un less b oth endp oints i and j are in the same set. On the other hand y ij cannot b e equal to 1 u n less b oth end p oints i and j are in S . The sum constraint ensur es that at least one edge is in the segmen t S and at least one edge is in the complemen t - the bac kground . Otherwise the ratio is n ot b e d efined in the fi rst case and the optimal s olution is to c ho ose S = V in the s econd. W e remo ve the sum constrain t from th e form ulation and replace it by setting for some edge in the ob ject to serve as “se ed” and some edge in the bac kground to serve as “seed”. Adding seeds is an appr oac h ofte n used in segmenta tion, see e. g. [3] w here the user k eeps adding “seeds” until th e bip artition is satisfactory . Here the choi ce is m ad e once, and can b e replaced b y en umerating the p ossible pairs of edges that serve as ob ject and bac kground edges. Since for b oth the ob j ect and it complemen t the cut v alue is the same, the solution will alw a ys b e in terms of the larger segmen t in the bipartition that is likel y to conta in higher total similarit y weig hts. Th e edge that w e set to b e in the source is therefore usually the one in th e bac kgrou n d and the one in the sink would b e in the ob ject. W e th u s replace the sum constraint by setting y i ∗ j ∗ = 1 and y i ′ j ′ = 0 for some pair of edges in the bac kground and the ob ject resp ectiv ely . Once th e sum constrain t has b een r emo v ed , the pr oblem formulat ion is easily recognized as a monotone inte ger programming with up to three v ariables p er inequ alit y according to th e definition pr o vided in Ho c hbaum’s [16]. An y suc h p roblem was sh o wn there to b e solv able as a minim u m cut p roblem on a certain asso ciated graph. Because the ob jectiv e fu nction is a ratio, w e first “linearize” the problem. 5.1 Linearizing the ob jective function The λ - question for the n ormalized cut problem is: Is there a feasible subset V ′ ⊂ V such that P [ i,j ] ∈ E w ij z ij − λ P [ i,j ] ∈ E w ′ ij y ij < 0? One p ossib le approac h is to utilize a binary searc h pro cedu re that calls for the λ -question O (log ( U F )) times in order to solv e the problem, wh ere U = P [ i,j ] ∈ E w ij , and F = P [ i,j ] ∈ E w ′ ij for the w eigh ts at the denominator w ′ ij . 8 With the construction of the graph w e observe that one can u se instead a parametric ap- proac h w hic h is significantly more efficient. W e note that the λ -qu estion is the f ollo w ing mono- tone optimization p roblem, ( λ -NC) min P [ i,j ] ∈ E w ij z ij − P [ i,j ] ∈ E λw ′ ij y ij sub ject to x i − x j ≤ z ij for all [ i, j ] ∈ E x j − x i ≤ z j i for all [ i, j ] ∈ E y ij ≤ x i for all [ i, j ] ∈ E y ij ≤ x j y i ∗ j ∗ = 1 and y i ′ j ′ = 0 x j binary j ∈ V z ij , y ij binary j ∈ V . If the optimal v alue of this pr oblem is negativ e th en the answer is y es, otherwise the answer is no. This pr oblem is an int eger optimization problem on a totally un imo dular constrain t matrix. Th at means that we can solve the linear p rogramming r elaxation of this problem and get a basic optimal solution that is inte ger. Instead we will use a muc h more efficien t algorithm describ ed in [16] w hic h relies on the monotone pr op erty of th e constrain ts. 5.2 Solving the λ question with a minim um cut pro cedure W e constru ct a directed graph G ′ = ( V ′ , A ′ ) with a set of n o des V ′ that has a n o de for eac h v ariable x i and a no de for eac h v ariable y ij . T h e no des y ij carry a negativ e we ight of − λw ij . The arc fr om x i to x j has capacit y w ′ ij and so do es the arc f rom x j to x i as in our problem w ij = w j i . The t wo arcs from eac h edge- n o de y ij to th e endp oin t no des x i and x j ha ve infi nite capacit y . Figure 1 sho w s the basic gadget in the graph G ′ for eac h edge [ i, j ] ∈ E .  x i x j y ij w ij - Ȝ w’ ij w ji  Figure 1: Th e basic gadget in th e graph rep resen tation. W e claim that an y fi nite cut in this graph, that has y i ∗ j ∗ on one s id e of th e bipartition and y i ′ j ′ on the other, corresp onds to a f easible s olution to the problem λ -NC. Let the cut ( S, T ), where T = V ′ \ S , b e of finite capacit y C ( S, T ). W e set the v alue of th e v ariable x i or y ij to b e equal to 1 if the corresp ondin g n o de is in S , and 0 otherwise. Because the cu t is fin ite, then y ij = 1 implies that x i = 1 and x j = 1. 9 Next we claim that for an y finite cut the sum of the w eights of the y ij no des in the source set and the capacit y of the cut is equal to the ob jectiv e v alue of problem λ -NC. Notice that if x i = 1 and x j = 0 then the arc from the no de x i to no de x j is in the cut and ther efore the v alue of z ij is equal to 1. W e n ext create a source no de s and connect all y ij no des to the source with arcs of capacit y λw ′ ij . The n o de y i ∗ j ∗ is then shr unk with a source n o de s and therefore also its endp oin ts no des. The n o de y i ′ j ′ and its endp oin ts no d es are shrunk with the sink t . W e denote this graph illustrated in Fig u re 2, G ′ st . Source edge s t x 1  Ȝ w' 23 x 2 x 4 x 3 x 5 y 12 y 23 y 34 y 45           Ȝ w' 34 Ȝ w' 45 Sink edge Figure 2: Th e graph G ′ st with edge [1 , 2] as source seed and edge [4 , 5] as sink seed. Theorem 5.1 A minimum s, t - cut in the gr aph G ′ st , ( S, T ) , c orr esp onds to an optimal solution to λ -NC by setting al l the variables whose no des fal l in S to 1 and zer o otherwise. Pro of: Note that w henev er a no d e y ij is in th e s in k set T the arc connecting it to the s ource is included in the cut. L et the set of x v ariable no des b e d enoted b y V x and the set of y v ariable no des, excluding y i ∗ j ∗ , b e denoted b y V y . Let ( S, T ) b e any fin ite cu t in G ′ st with s ∈ S and t ∈ T and capac ity C ( S, T ). C ( S, T ) = X y ij ∈ T ∩ V y λw ′ ij + X i ∈ V x ∩ S,j ∈ V x ∩ T w ij = X v ∈ V y λw ′ v − X y ij ∈ S ∩ V y λw ′ ij + X x i ∈ V x ∩ S,x j ∈ V x ∩ T w ij = λW ′ + [ X i ∈ V x ∩ S,j ∈ V x ∩ T w ij − X y ij ∈ S ∩ V y λw ′ ij ] . This p ro ves that for a fi xed constant W ′ = P v ∈ V y w ′ v the capacit y of a cut is equal to a constan t W ′ λ p lus the ob jectiv e v alue corresp onding to the f easible s olution. Hence the partition ( S, T ) m in imizing the capacit y of th e cut minimizes also the ob jectiv e function of λ -NC. 10 5.3 A parametric pro c edure for solving normalized cut The source adjacen t arcs in the graph G ′ st are monotone increasing with λ . As the v alue of λ increases th e source set of the resp ect ive min im um cuts are nested. This is called the n estedn ess lemma. Although the capaci ty of th e cut is increasing with an y increase in λ the set of no des in the sour ce s et can th us b e incremen ted only n ′ = | V ′ | times. W e call the v alues of λ where the source set expands by at least one no d e, the br e akp oints of the parametric cut. Let the breakp oints b e λ 1 > λ 2 > . . . > λ ℓ , with corr esp onding minimal source sets, S 1 ⊂ S 2 ⊂ . . . ⊂ S ℓ . As a result of the nestedness lemma ℓ ≤ n ′ for a graph on n ′ no des since there can b e no more than n ′ differen t nested source sets. The capacit y v alue of th e minimum cut is increasing as a function of increasing v alues of λ along a p iecewise linear conca v e curv e. Theorem 5.2 Al l br e akp oints of the density gr aph c an b e found by solving a p ar ametric min- imum cut pr oblem wher e the sour c e adjac ent c ap acities of ar cs ar e line ar functions of the p a- r ameter, λ . Gallo, Grigoriadis and T arjan sho wed in [9] ho w to fi n d all the b reakp oints and the cor- resp ond ing min im um cuts in the same complexit y as that required to solv e a single minim u m s, t -cut problem with th e pu sh-relab el algorithm of [11]. The pseudoflow algorithm for maxi- m um flo w and min im um cut (see Ho ch baum [14]) also fin ds the p arametric b reakp oints in the complexit y of a sin gle min im um s , t -cut. Once all the b r eakp oint s are generated, we searc h f or the largest v alue of λ among the b reakp oints so that the optimal v alue of λ -NC is n egativ e, or equiv alen tly , the min imum s, t -cut v alue that is strictly less than λW ′ . T o summarize, let T ( n, m ) b e the ru nning time required to solv e the minimum cut problem on a graph with n no des and m arcs. In the graph G ′ st the num b er of no des is n ′ = n + m where m is the num b er of adj acencies or edges in the image graph. The num b er of arcs in G ′ st , m ′ = | A ′ | , is O ( m ). F or general graph this run ning time is O ( m 2 log m ) w ith either the pseudoflow algorithm or the p u sh-relab el algorithm. The degree of eac h n o de is constan t for imaging applications so for that conte xt m ′ = O ( n ) and n ′ is O ( n ) and the running time is O ( n 2 log n ). Theorem 5.3 The normalize d cut pr oblem i s solva ble i n the running time of a minimum s, t - cut pr oblem, T ( n ′ , m ′ ) . Remark: It ma y b e desirable to solve ( λ -NC) without sp ecifying a source and a sink. The problem is then to p artition the graph G ′ st to t wo n onempt y comp onen ts so that the cut s ep arat- ing them is minim u m. This problem is the directed min im um 2-cut p roblem. It was sho wn b y Hao and O rlin [13], that the directed minimum 2-cut prob lem is solv ed in the same complexit y as a single minimum s, t cut p r oblem, with the push -r elab el algorithm. (This was sh own to hold also for the pseudoflow algorithm.) In order to solv e the normalized cut problem, the algo r ith m pr o duces a sequence of nested solutions for all p ossible v alues of the p arameter λ . Eac h suc h solution repr esen ts a different w eight in g of th e cut ob jectiv e v ersu s the s imilarit y ob jectiv e. As the v alue of the λ gro ws the similarit y ob j ectiv e is more pr omin en t and the optimal s olution S λ expands. Although the normalized cut r atio pr oblem’s optimal solution comprises of a single connected comp onent (see part I I), the sequen ce of optimal solutions to the range of parameter v alues is not necessarily 11 formed of a single connected comp onen t. Su c h solutions could b e more meaningfu l in medical images for instance, where lesions are the features sough t, but they often app ear as d isjoin t comp onent s in the image. 6 A sk etc h of the tec hnique for Ratio Regions The w eigh ted ratio regions problem, once linearized, is an in stance of the s -excess p roblem in [14]. As b efore we formulat e the problem fi rst. Let x i = ( 1 if i ∈ S 0 if i ∈ ¯ S . Let z ij = 1 if exactly one of i or j is in S . z ij = ( 1 if i ∈ S , j ∈ ¯ S , or i ∈ ¯ S , j ∈ S 0 if i, j ∈ S or i, j ∈ ¯ S . Let the similarit y wei ght on eac h edge b e w ij and the weig ht of nod e (pixel) j b e v j . With these parameters the r atio regions p roblem formulatio n is, (RR) min P w ij z ij P v j x j sub ject to x i − x j ≤ z ij for all [ i, j ] ∈ E x j − x i ≤ z j i for all [ i, j ] ∈ E x j binary j ∈ V z ij binary j ∈ V . The corresp onding λ -question is, ( λ -RR) min P [ i,j ] ∈ E w ij z ij − P j ∈ V λv j x j sub ject to x i − x j ≤ z ij for all [ i, j ] ∈ E x j − x i ≤ z j i for all [ i, j ] ∈ E x j binary j ∈ V z ij binary j ∈ V . The graph constructed for that problem is of the s ame size as the original graph G . Each no de r ep resen ting a v ariable x j has an arc going to sin k n o de with capacit y λv j . One v ariable no de, x s is selected arb itrarily as corresp onding to a source “seed”. The grap h G ′ st has O ( n ) no des and O ( m ) arcs. The alg orithm solving the p roblem is then a s imple p arametric cut algorithm in that graph, with run time T ( n, m ). F urthermore, the parametric s, t -cut algo r ithm deliv ers the sequence of optimal nested solutions for all v alues of λ , as well as the optimal solution to the ratio p roblem, in r un time T ( n, m ). 12 x i x j w ij w ji Ȝ v i s  x s w jp w pj Ȝ v j Ȝ v p Ȝ v s x p t Figure 3: Th e graph G ′ st for the ratio regions problem with no d e x s serving as s ource seed. 7 Brief remarks ab out the other problems The problem n ormalized cut’ is: m in S ⊂ V C ( S, ¯ S ) C ( S,V ) . This problem is in fact identica l to the nor- malized cut problem. T o see that notice th at C ( S, V ) = C ( S, S ) + C ( S, ¯ S ). S ubstituting this w e get : C ( S, ¯ S ) C ( S, V ) = C ( S, ¯ S ) C ( S, S ) + C ( S, ¯ S ) = 1 1 + C ( S,S ) C ( S, ¯ S ) . This quantit y is minimized w hen C ( S,S ) C ( S, ¯ S ) is maximized. T h u s the same alg orithm applies. Concerning the maxim um density problem, it is p resen ted as a minim u m s, t -cut problem on an u n b alanced b ipartite graph w ith n o des represen ting the ed ges of th e graph G on one sid e of the b ipartition and no d es representing the n o des of G on th e other. (Detail s are a v ailable in [17].) That b ipartite graph has m + n n o des, and m ′ = O ( m ) arcs. The complexit y of a s ingle minim u m s, t -cut in suc h graph is therefore O ( m 2 log m ). Th is how ev er can b e imp ro ved. The num b er of iterations required by the p ush-relab el algorithm or the pseud oflo w algorithm is b ounded b y a function of the length of the longest residual p ath in the graph – O ( m ′ n ′ ) – where m ′ is the num b er of arcs in the bipartite graph and n ′ is the maximum residual p ath length. In the λ -net work constructed for the λ -question, this length n ′ is at most 2 n + 2 as eac h path alternates b etw een the t w o sets in the partition. This fact is used by Ahuja, O r lin, Stein and T arjan, [1], who devised imp ro ved p ush-relab el algorithms f or u n b alanced bipartite graphs. Amo n g those, the most efficient for parametric minim u m cut is an adaptation of the p arametric pu sh-relab el algorithm of Gallo, Grigoriadis and T arjan with r un time O ( m ′ n ′ log( n ′ 2 m ′ + 2)). This run time trans lates to O ( mn log( n 2 m + 2)) for the parametric p r oblem solving the minimum density p roblem on a general graph, improving on the O ( m 2 log m ) complexit y for a d irect application of the p arametric cut alg orithm. 13 Figure 4: The input to the normalized cut p ro cedur e on th e left. The output, on the right, segmen ted and s ep arated the feature fr om the bac kground . 8 Exp erimen tal example The normalized cut pro cedure wa s implemen ted u sing the pseudoflow algorithm, [14], which solv es the minim u m s, t -cut problem (and the maximum fl o w problem.) The pseudoflow algo- rithm is fast in theory an d in practice, [4]. The co de and its p arametric v ersion are av ailable for do w nload at h ttp://riot.ieo r .b erkele y .edu /riot/Applications/Pseudoflo w/ . The algorithm describ ed here w as applied to a synt h etic image pro vided in Figure 4. T h e optimal solution d ifferen tiated precisely the feat u re from th e backg r ound. References [1] R. K. Ahuja, J. B. Orlin, C. Stein and R. E. T arjan. Imp ro ved algorithms for bipartite net wo r k fl o w. SIAM Journal of Computing , 23: 5 1994, 906–933 . [2] Blak e A. and A. Zisserman. Visual r e c onstruction . MIT Press, 1987. [3] Bo yko v Y. and J olly M-P . Int er activ e Graph Cuts for Optimal Boundary & Region Segmen- tation of Ob jects in N-D Images. International Confer enc e on Computer Vision, (ICCV) , v ol. I, p p. 105–112, 2001. [4] B. Chandran and D. S. Hoch baum. A Compu tational Study of the Pseudoflo w and Push- relab el Algorithms for the Maxim um Flo w Problem. Dec 2005. O p er ations R ese ar ch , T o app ear. [5] J. Cheeger. A Lo wer Bound for the S mallest Eigen v alue of th e Laplacian. Pr oblems in Ana lysis , R.C. Gunning, ed., pp. 195–199, Princeton Univ. Pr ess, 19 70. [6] F.R.K. Ch ung. Sp e ctr al Gr aph The ory . Am. Math. S o c., 1997. 14 [7] I. J . Co x, S. B. Rao, Y. Zh ong. ’Ratio Regions’: A T echnique f or Image Segmen tation. Pr o c. Int. Conf. on Pattern R e c o gnition , B:5 57-564, August 1996. [8] W. H. Cu nningham. Optimal attac k and reinforcement of a net wo r k. J . ACM 32, 549-561, 1985. [9] G. Gallo, M. D. Grigoriadis and R. E. T arjan. A fast parametric maximum flo w algorithm and applications. SIA M Journal of Comp u ting , 18:1, 1989 , 30–55. [10] A. V. Go ldb erg, Finding a maxim um density subgraph, UC Berk eley rep ort UCB/CSD/84/1 71, 1984. [11] A. V. Goldb erg and R . E. T arjan. A New Ap proac h to the Maxim um Flo w Problem. Journal of th e ACM , 35, 921–940, 198 8. [12] O. Goldschmidt and D. S. Ho ch baum. A Polynomial Algorithm for the K-Cut Problem. Mathematics of Op e r ations R ese ar ch , 19:1, 24– 37, 1994. [13] J. Hao and J . B. Orlin. A faster algorithm for fi nding the min imum cut in a dir ected graph. J. A lgorithms , 17: 3, 1994, 424–446 . [14] D. S. Ho ch baum. The Pseudoflow algorithm: A n ew algorithm for the maximum flo w problem. Op er ations R ese ar c h to app ear. Extended abstract in The pseud oflo w algo r ithm and the pseudoflo w-based simplex for the maxim um flo w p roblem. Pro cdeedin gs of IPCO 98, June 1998. LNCS , 1412, 325– 337. [15] D. S. Ho c hbaum. An efficient algorithm for image segmen tation, Mark ov random fields and related p roblems. Journal of the A CM , 48:4, 686 –701, 2001. [16] D. S. Ho c hbaum. Solving integ er programs ov er m onotone inequalities in three v ariables: A framew ork for half integ rality and go o d appro ximations. Eur op e an J ournal of Op er ationa l R ese ar ch 14 0:2 , 2002, 291 –321. [17] D. S. Ho ch baum. Dynamic evol u tion of economically preferred facilities. Eur op e an Journal of Op er ational R ese ar ch to app ear. UC Berk eley manuscript, 2005. [18] S. Sark ar and K .L. Bo yer. Quan titativ e Measures of Ch an ge Based on F eature Organi- zation: Eigen v alues and Eigen vect ors . Pr o c. IEE E Conf. Computer Vision and Pattern R e c o gnition , 478 , 1996. [19] Shi, J. and Malik, J. 2000. Normalized cuts and image segmen tation. IEEE T r ans. Pattern Ana l. Mach. Intel l. , 22:8, 888–905. [20] E. Sh aron, M. Galun, D. Sharon , R. Basri and A. Brandt. Hierarc h y and adaptivity in segmen ting visual s cenes. Natur e 442, 810 –813, 17 August 2006. [21] Da vid A. T olliv er and Gary L. Miller. Graph Partiti on in g by Sp ectral Round ing: Applica- tions in Image Segmenta tion and Clu stering. CVPR ’06 , pages 10 53–1060, 2006 . [22] S. W ang and J.M. S iskind. Im age Segmenta tion with Ratio Cu t. IEEE T r ansactio ns on Pattern A nalysis and Machine Intel ligenc e , 25:6, 675–69 0, 2003. 15