Locked constraint satisfaction problems

Lo c k ed constrain t satisfaction problems Lenk a Zdeb orov´ a 1, 2 and Mar c M´ ez ard 1, 2 1 Universit ´ e Paris-Sud, LPTMS, UMR8626, Bˆ at . 100, Uni versit´ e Paris-Su d 91405 Orsay c e dex 2 CNRS, LPT M S, UMR8626, Bˆ a t. 100, Uni versit ´ e Paris-Sud 91405 Orsay c e dex (Dated: Octob er 25, 2018) W e introdu ce and study the random ‘locked’ constraint satisfaction problems . When increasing the density of constrain ts, t h ey displa y a broad ‘clustered’ phase in which the space of solutions is divided into many isolate d p oin ts. While the phase diagram can b e found easily , these problems, in their clu stered ph ase, are extremely hard fro m the algori thmic p oint o f view: the best known algorithms all fail to find solutions. W e thus prop ose new b enchmarks of really hard optimization problems and pro vide insigh t in to the origin of their t ypical hardness. P ACS n umbers: 89.70.Eg,75.10.Nr,64.70.P- Constraint satisfaction pr o blems (CSPs) are one of t he main build ing blocks o f complex systems studied in com- puter science , infor mation theory and sta tistical physics. Their wide range o f applicability arises from their very general nature: given a s et of N discrete v aria bles sub ject to M constra in ts, the CSP c onsists in deciding whether there exists a n a s signment of v ar iables which satisfies si- m ultaneously all the constraints. In computer science CSPs are at the core of co mputational complexity stud- ies: the satisfiability of bo ole an formulas is the canoni- cal example o f an intrinsically ha rd, NP-complete, prob- lem [1]. In information theory error correcting co des a lso rely on CSPs. The tra nsmitted information is enco ded int o a co deword sa tisfying a set of constra in ts, so that information may be retrieved after transmissio n through a noisy channel, using the knowledge of the constra ints. Many other practica l problems in scheduling a collection of tas k s or in har dw are and softw are verification and test- ing a re viewed as CSPs. In statistica l ph ysics the inter- est in CSP s stems from their clo se relation with theory of spin glas s es. Answering if frustr ation is av oidable in a system is a first, and sometimes highly nontrivial, step in under standing its low temp erature b ehavior. Metho ds of statistica l physics provide p ow erful to ols to study statistical prop erties o f CSP s [2, 3 ]. The mean field approach is known to b e exac t if the underlying graph of constraints [4] is either fully connected or lo cally tree- lik e. It also has alg orithmic, a nd pr actical, conseque nc e s : in contrast with the usual situation in physics, CSPs o n a lo cally tr ee-like gr aph ar e used in practice, for instance in low density parity chec k co des [5 ], which a re among the b est erro r-corr ecting co des aro und. Many CSP s are NP-co mplete. Nevertheless, lar ge classes of instances can b e easy to solve. It is one of the main g oals of theor etical computer science to under- stand why some instances are har der than others , wher e the har dness comes fro m and how to av oid it, b eat it or use it. The ra ndom K -satisfiability ( K -SA T) problem where clauses are chosen uniformly at random betw een all po ssible one s has play ed a prominent role in appr oach- ing this goal. In r andom K -SA T there exists a sharp satisfiability thre shold. This is a pha se transition p oint separating a ’SA T’ pha se with low density of cons tr aint s where instance s are almost alwa ys sa tisfiable, from a n ’UNSA T’ phase wher e, with high probability , there is no solution to the CSP [6, 7]. The hardest instances lie near to this thresho ld [8, 9]. The main insight came from sta - tistical physics studies [10, 11, 1 2, 1 3, 14, 15, 16] which allow to des cribe the s tructure of the space of solution of the random K -SA T problem. The most in teresting re- sult is the existence of an intermediate “clustered” phase, just b elow the SA T-UNSA T thresho ld, whe r e the space of solutions splits into well separated clusters . A ma- jor o pen question consists in under standing if and how the existence of clusters makes the problem ha rder. The survey propa g ation a lgorithm, which explicitly tak es into account the clusters , is the b est known so lver very close to the SA T-UNSA T thre s hold [1 3], but some lo cal search algorithms also p erform well inside the clustere d phase [17, 18]. Another prop osition, put forward in [15], is that solutions in clusters with fr ozen v ariables, ta king the same v a lue in the whole cluster, are har d to find. It was sho wn in [19] tha t, even if solutions belo nging to clus- ters without frozen v a riables are exp onent ially rare , s ome message pas sing algorithms may b e able to find them. In this letter we intro duce and study a broad class of CSPs which are extremely frozen problems: all the clusters consists of a single configura tion, th us all the v ar ia bles are fr o zen in every cluster. W e show tha t these problems are extremely difficult from an a lgorith- mic p oint of view: all the be st known alg orithms fail to solve them in this clus tered phase. At the same time the description of their phase diagr a m can b e car ried out in details with re la tiv ely s imple statis tica l physics metho ds. Definition – W e define an o c cup ation CSP over N bi- nary v a riables, s 1 , . . . , s N ∈ { 0 , 1 } a s follows: each con- straint a connects to K r a ndomly chosen v ariables, and its sta tus dep ends on the sum r of these v ariables. The constraint is characterized b y a ( K + 1) comp onent vector A = ( A 0 A 1 · · · A K ), with A r ∈ { 0 , 1 } : it is sa tisfied if and only if A r = 1. W e shall study her e ho mogeneous mo dels in which a ll constraints connec t to the sa me num b er K 2 of v ariables, and a r e characterized by the sa me vector A . According to [20] the o ccupation CSPs ar e NP -complete if K > 2, A 0 = A K = 0 and A is not a parity chec k. The lo cke d o c cup ation pr oblems (LOP) are o ccupation CSPs satisfying t wo co nditions: (a) ∀ i = 0 , . . . , K − 1 the pro d- uct A i A i +1 = 0, (b) a ll v ar iables a re present in at leas t t wo co nstraints. Simple examples o f LOPs ar e po sitive 1-in-3 satisfia bility [21], A = 01 00, or parity chec ks [5], A = 01010 , on gr aphs witho ut leav es. In or der to go from one solution (satisfying assignment) of a LOP to another one, it is necessa ry to flip at least a closed lo o p of v ar iables in the facto r gr aph represe ntation of [4]. This stays at the ro o t of the crucial prop erty that clusters ar e po in t-like and separ ated by an ex tens iv e distance w he n the density of constra in ts is large enoug h (ab ov e l d ). In o rder to fully characterize a ra ndom LOP ensemble, one needs to define the degre e distribution of v ariables. W e will study here tw o ensembles. The reg ula r ense mble, where every v ariable a pp ear s in exactly L constra in ts, and the trunca ted Poisson ensemble with degree distribu- tion Q (0) = Q (1) = 0 , Q ( l ) = e − c c l /l ![1 − (1 + c ) e − c ] , l ≥ 2 and av erag e connectivity l = c (1 − e − c ) / [1 − (1 + c ) e − c ]. Phase diagr am – Denoting by a, b , . . . the indices of constraints and i, j, . . . tho se of v ar iables, the b elief prop- agation (BP) equa tions [22] ar e given by: ψ a → i s i = 1 Z a → i X { s j } δ ( A s i + P j s j − 1) Y j ∈ ∂ a − i χ j → a s j , (1) χ j → a s j = 1 Z j → a Y b ∈ ∂ j − a ψ b → j s j , (2) where ∂ a are a ll the v a riables app earing in constraint a , and ∂ i a ll the c o nstraints in which v ariable i appea rs. χ j → a s j is the probability that spin j takes v a lue s j when a was r emov ed from the graph, and Z are no rmalization constants. The BP entrop y (the lo g arithm of num b er of configuratio n sa tisfying all c o nstraints, divided by N ) is s = 1 N X a log ( Z a + ∂ a ) − 1 N X i ( l i − 1) log ( Z i ) , (3) where: Z a + ∂ a = X { s i } δ ( A P i s i − 1) Y i ∈ ∂ a Y b ∈ ∂ i − a ψ b → i s i ! , (4 ) Z i = Y a ∈ ∂ i ψ a → i 0 + Y a ∈ ∂ i ψ a → i 1 . (5) In order to find a fixed p oint of eqs. (1-2) and compute the quenched av erage o f the en tropy we use the p opulation dynamics technique [2], with p opulation sizes of order 1 0 4 to 10 5 . It turns out that this pro cedure alwa ys con verges to the same fixed p oint. The phase diagram of LOPs is muc h simpler to a na lyze than the one o f general CSPs , and can b e deduced pur ely from the BP ana lysis. This is due to the fact that, in the clustered phase, every cluster reduces to a single is o lated configuratio n. The survey pro pagation (SP) equa tions [13] a r e then g reatly simplified. Their iteration either leads to a trivial fixed p oint, w he r e every v ar ia ble is in the so called ” joker” state [2 3], o r to a fixed p oint where no v ariable is in the ”joker” state. In this sec o nd case the SP equatio ns reduce to the BP eqs. (1 - 2), a nd the complexity function (logarithm of num ber of cluster s ) is equal to the ent ropy (3), in agr eemen t with the p oint-lik e nature o f clusters . The c lustered phase is then identi- fied fr om the iter ative stability of this second, non-trivial fixed p oin t. It is itera tiv ely stable when the av erage con- nectivity is ab ov e a threshold: l > l d , while the regime l < l d corres p onds to a ’liquid’ phase. The intuitiv e dif- ference b etw e e n the tw o phases is that in the clustered phase o ne has to flip an extensive num b er of v ariables to go from one solutio n to another, while in the liquid phase the addition of any infinitesimal temper ature is enough to b e able to connect all solutions. The sa tisfiabilit y threshold l s is defined a s follows: If the average connectivity is l < l s then a satisfying a s- signment almost sur ely ex ists (in N → ∞ ), a nd if l > l s then ther e is almost surely no sa tisfying configur ation. In LO Ps we ca n find l s as the a verage co nnec tiv ity a t which the RS entropy (3 ) b ecomes z ero. T able. I gives the v alues o f cluster ing and satisfiability thresholds for the no n- trivial LOPs with K ≤ 5. A name L s l d l s 0100 1-in-3 3 2.256(3) 2.368(4) 01000 1-in- 4 3 2.4 42(3) 2.657(4 ) 00100* 2-in-4 3 2.513 2.827 01010* od d 4-PC 4 2.856 4 010000 1-in-5 3 2.594(3) 2.901(6) 001000 2-in-5 4 2.690(3) 3.180(6) 010100 1-or-3-in-5 5 3.0 68(3) 4.724(6 ) 010010 1-or-4-in-5 4 2.4 08(3) 3.155(6 ) T A BLE I: The clustering l d and satisfiability l s thresholds in the lo ck ed o ccupation problems for K ≤ 5 in th e truncated P oisson ensemble. In the regular ensemble L s is the first un- satisfiable or critical connectivity , the first clustered case is L d = 3. The error bars originate in th e statistical nature of the p opulation dy namics technique. Symmetric LOPs where the satisfiabilit y threshold can b e computed analytically are indicated by *. When a LOP is symmetric, i.e., A r = A K − r for a ll r = 0 , . . . , K , and this 0 − 1 s ymmetry is no t sp onta- neously bro ken, the sa tisfiability thre shold can b e com- puted rig o rously using the 1 st and the 2nd moment meth- o ds: The annealed entrop y h Z i ≡ ex p ( N s ann ) is: s ann ( l ) = log 2 + l K log " 2 − K K X r =0 δ ( A r − 1)  K r  # . (6) 3 By computing the se c ond moment h Z 2 i and using the Chebyshev’s inequality , as in [24, 25], w e have shown that the annealed e ntropy is equal to the t ypical one, thus the satisfiability threshold l s is given by s ann ( l s ) = 0. Exam- ples of LOPs for which this works a re the parity chec ks A = 01010 , as well a s A = 00 1 00 , 000 1000 , 0010100, etc. Note that fo r instance A = 01001 0 do es not belo ng to this class b ecaus e its 0 , 1 symmetry is sp ontaneously bro k en. Algo rithms – W e attempt to find solutio ns to LO Ps in their satisfia ble phase using thre e algor ithms which are among the b est for hard random instances of the K-satisfia bility problem: belief propagatio n decima tio n (BPd) [14] (which is the same a s survey propagation [13] in LO Ps), s tochastic lo cal sea rch (SLS) [26], and rein- forced b elief propag ation (rBP) [2 7]. In BPd one use s the knowledge of marg inal v ariable probabilities from BP equations in or der to identify the most biase d v ariable, fix it to its most probable v a lue, and reduce the problem. In K -SA T the SP decimation (whic h in LOPs is equiv ale n t to BP d) has b een shown to be very efficient, on very large problems, even very near to the satisfiability threshold [13]. Ho wev er, in LOP s the BP decimation fa ils badly . F or ex ample in the 1-or - 3-in-5 SA T problem, on truncated Poisson graphs with M = 2 · 10 4 constraints, the pro babilit y of success is a bo ut 25% at l = 2 , and less than 5 % a t alrea dy l = 2 . 3 , wa y below the clustering threshold l d ≃ 3 . 0 7. Although w e do not know how to ana lyze directly the BPd pro cess, some mechanisms ex plaining the failure of the decimation strategy can b e understo o d using the ap- proach of [28]. The idea is to analyze a slightly simpler decimation pro cess, where the v aria ble to be fixed is cho- sen uniformly at r andom and its v a lue is chosen acco rd- ing to its exact ma rginal pro babilit y , which is assumed to be a ppr oximated by BP . The reduce d for m ula after θN s teps is equiv alent to the reduced for m ula cre a ted by cho osing a solution uniformly at rando m and reveal- ing a fraction θ of its v ariables. The num b er of v ari- ables which were either revealed or are dir ectly implied by the revealed ones is denoted Φ ( θ ). The p erformance of this ‘unifor m’ BP decimation can b e under sto od from the s hape of the function Φ( θ ), which we hav e computed from the cavit y metho d. In Fig. 1 we show that the theoretical cur ve Φ ( θ ) agrees with numerical results in r egular 1-o r-3-in-5 SA T. A t connectivity L = 3 the function has a discontin uit y at θ s ≃ 0 . 46 , thu s after fixing a fraction θ s of v ariables an infinite av ala nc he of direct implications follows and small error s in the BP es timation of marg ina ls lead to a con- tradiction with hig h pr obability . A t co nnectivit y L = 2 the function Φ( θ ) → 1 at θ 1 ≃ 0 . 7 3. This means that if a fraction θ > θ 1 of v ariables in a random solution is re - vealed the r e sidual problem has only this single solution. An y mistake in the previously fixed v a r iables matters and causes a co n tradiction. In all the LOP s we hav e studied, Φ( θ ) has one these tw o fatal prop erties. The inset o f 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 variables implied variables revealed L=2 theory L=2 BP dec. L=3 theory L=3 BP dec. 0 0.1 0.2 0.3 0.4 0 0.1 0.2 0.3 0.4 0.5 L=2, max L=2, unif L=3, max L=3, unif FIG. 1: U niform BP decimation in regular 1-or-3- in-5 SA T with L = 2 and L = 3: plot of Φ( θ ) , as obtained analytically (lines) and from t h e u niform BP decimation (p oints): the tw o plots agree p erfectly . F or L = 3 the decimation fails b ecause of av alanches at the discon tinuit y of Φ( θ ), for L = 2 it fails when Φ( θ ) → 1 for θ < 1. Inset: Comparison b etw een BPd and un iform BP decimation. The number of directly implied v ariables is plotted against num b er of vari ables which were free just b efore fixing them. The tw o metho ds are very close, and th ey fail at abou t t he same v alue of θ . Fig. 1 shows that, in LOP s, there is not muc h difference in the b ehaviors of BPd and this uniform BP decima tio n. Sto c hastic lo cal search (SLS) a lg orithms exist in many different versions and are used in most pra c tica l ca ses where the exha us tiv e search is to o time consuming . The main idea of the fa mily of algorithms is to pe r form a ran- dom walk in configura tio nal spa ce, trying to minimize the the num b er of unsatisfied cons tr aint s. In the implemen- tation of [17 ], a v a riable which b elong s to at least one unsatisfied constra in t is chosen randomly . If flipping this v ar ia ble do es not increa se the ene r gy , the flip is accepted. If it increas e s the energ y , the flip is a ccepted with prob- ability p . This is rep eated until either one finds a s olu- tion, or the num ber of steps per v ar iable exceeds T . The parameter p must b e optimized. In Fig. 2 we plot the fraction of successful runs for the 1-or -3-in-5 SA T with M co nstraints and p = 0 . 00003. Even with the la r gest v alue of T we have not b een a ble to solve instances with av erage connectivity la rger than 3 . 05. The b elief propaga tion r einforcement (r BP) was orig- inally introduce d in [27]. The main idea is to add an external field µ i s i which biases the v ariable i in the di- rection o f the marginal pro babilit y co mputed from the BP messages. This mo difies BP eq . (2) to ψ i → a s i = µ i s i Q b ∈ ∂ i − a ψ b → i s i / Z i → a . The algo rithm then works as follows: Iter ate the BP equatio ns n - times. Up date all the external fields: If ξ i 1 < ξ i 0 set µ i 1 = π l i , µ i 0 = (1 − π ) l i , otherwise set µ i 1 = (1 − π ) l i , µ i 0 = π l i , where ξ i s i = ( µ i s i ) 1 / ( l i − 1) Q a ∈ ∂ i ψ a → i 1 . A t each iteration one chec ks if the most probable co nfiguration, given by s i = 0 if 4 0 0.2 0.4 0.6 0.8 1 2.9 2.95 3 3.05 3.1 3.15 3.2 3.25 probability of success average connectivity l d rBP M=2.10 4 T=5.10 4 rBP M=2.10 4 T=5.10 3 rBP M=4.10 3 T=5.10 3 SLS M=2.10 4 T=5.10 5 SLS M=2.10 4 T=5.10 4 SLS M=4.10 3 T=5.10 4 FIG. 2: P erformance of reinforced BP and sto chastic local searc h for 1-or- 3- in-5 SA T with M constraints. The fraction of successful ru ns is plotted against the avera ge connectivity l . The clustering threshold l d is marked, and the satisfiabilit y transition is at l s = 4 . 72. The maximal num b ers of steps p er v ariable T are chosen such th at the run ning times of rBP and SLS are comparable. µ i 0 > µ i 1 and s i = 1 otherwis e, is a s olution. If it is not one iterates a t maximum T times. W e chose n = 2 and optimized the v alue of π . In Fig . 2 we plo t the fraction of suc c e ssful runs for the 1 -or-3- in-5 SA T with π = 0 . 4 2 for 2 . 8 < l < 3 and π = 0 . 4 3 for 3 ≤ l < 3 . 2 . The p er- formance is mar ginally b etter than SLS, but a gain one cannot p enetrate into the c lus tered phase. W e hav e observed the same b ehavior for all LOP s we studied: the cluster ing tr ansition p oint l d seems to be a bo undary b eyond which all these three algo rithms fail. As shown in T able I, this p oint can be very far from the SA T-UNSA T tra nsition l s , meaning that ther e is a broa d range of instances wher e known algo rithms ar e totally in- efficient. The parity c heck problems are the exception as they ca n b e so lved with linear pro gramming algo rithms. Conclusions – LOPs make a br oad clas s o f extre mely hard constraint s atisfaction pro blems . Their phase dia- gram is simple: the set o f s a tisfiable configura tions b e- comes clustered when the av erage c o nnectivit y is l > l d , and it disapp ear s for l > l s . These tw o thresholds can b e computed efficiently using po pulation dynamics, and in the ca se o f some s ymmetric problems the v alue of l s can be confirmed r igorously . A t the same time, the b est al- gorithms known for random CSP fail to find so lutions in the cluster ed phase l d < l < l s . This difficulty is due to the ‘lo c ked’ natur e of the pro blem which reduces the clus - ters to single p oints. It will b e interesting to inv estigate if LOPs might b e used to design new efficient no nlinear error co rrecting co des, or if the pla n ted LO Ps a re go o d candidates fo r one-way functions in cry ptography . A cknow le dgment – W e thank F. K rzak a la, T. Mora and G. Semerjian for many fruitful discussio ns. [1] S. A. Co ok, in Pr o c. 3r d STOC (ACM , New Y ork, NY, USA, 1971), pp. 151–158. [2] M. M ´ ezard and G. Parisi, Eur. Phys. J. B 20 , 217 (2001). [3] M. M´ ezard and G. Pa risi, J. St at. Phys. 111 , 1 (2003). [4] F. R. Kschisc hang, B. F rey , and H.-A. Lo eliger, IEEE T rans. I nform. Theory 47 , 498 (2001). [5] R. G. Gallage r, IEEE T rans. Inform. Theory 8 , 21 (1962). [6] S. Kirkpatrick and B. Selman, Science 264 , 1297 (1994). [7] E. F riedgut, J. Amer. Math. So c. 12 (1999). [8] P . Cheeseman, B. Kanefsky , and W. M. T aylor, in Pr o c. 12th IJCAI (Morgan Kaufmann, San Mateo, CA, U SA, 1991), pp. 331–337. [9] D. G. Mitchell, B. Selman, and H. J. Levesque, in Pr o c. 10th AAAI (AAA I Press, Menlo Park, California, 1992), pp. 459–465. [10] R. Monasson, R. Zecchina, S. Kirkpatrick, B. Selman, and L. T roy ansky , Nature 400 , 133 (1999). [11] G. Biroli, R . Monasson, and M. W eigt, Eur. Phys. J. B 14 , 551 (2000). [12] M. M ´ ezard, G. Parisi, and R. Zecchina, Science 297 , 812 (2002). [13] M. M´ ezard and R. Zecc hina, Phys. R ev. E 66 , 056126 (2002). [14] F. Krzaka la, A. Montanari, F. R icci-T ersenghi, G. Se- merjian, and L. Zd eborov´ a , Pro c. N atl. Acad. Sci. 104 , 10318 (2007). [15] L. Zd eborov´ a and F. K rzak ala, Phys. Rev. E 76 , 031131 (2007). [16] A. Montanari, F. Ricci-T ersenghi, and G. S emerjian, J. Stat. Mec h. p. P04004 (2008). [17] J. Ardelius and E. A urell, Phys. Rev. E 74 , 037702 (2006). [18] F. Krzak ala and J. K urc han, Phys. R ev . E 76 , 021122 (2007). [19] L. Dall’Asta, A. Ramezanp our, and R. Zecchina, Phys. Rev. E 77 (2008). [20] T. J. Schaefer , in Pr o c. 10th STOC (ACM, New Y ork, NY, USA, 1978), p p. 216–226. [21] M. Garey and D. Johnson, Computers and i ntr actability: a guide to the the ory of NP-c ompl eteness (F reeman, S an F rancisco, 1979). [22] J. Y edidia, W. F reeman, and Y . W eiss, in Exploring Ar- tificial Intel ligenc e in the New Mil lennium (Science & T echnology Books, 2003), pp . 239–236. [23] A. Braunstein, M. M ´ ezard, M. W eigt, and R. Zecchina, in Computational Complexity and Statistic al Physics , edited by A. Percus, G. Istrate, and C. Mo ore (Ox ford Univer- sit y Press, 2003), p . 107. [24] M. M´ ezard, F. Ricci-T ersenghi, and R. Zecc hina, J. St at. Phys. 111 , 505 (2003). [25] S. Co cco, O . Dub ois, J. Mandler, and R. Monasson, Phys. Rev. Lett. 90 , 047205 (2003). [26] B. Selman, H. A. Kautz, and B. Cohen, in Pr o c. 12th AAAI (AAA I Press, Menlo P ark, CA, USA , 1994), pp. 337–343 . [27] J. Chav as, C. F urtlehner, M. M´ ezard, and R. Zecchina, J. Stat. Mec h. p. P11016 (2005). [28] A. Mon tanari, F. Ricci-T ersenghi, and G. Semerjian (2007), arXiv:0709.1667 v1 [cs.AI].