Perturbed Message Passing for Constraint Satisfaction Problems

Per turbed Messa ge P assing f or CSP P erturb ed Message P assing for Constrain t Satisfaction Problems Siamak Ra v anbakhsh mra v anba@ualber t a.ca Dep artment of Computing Scienc e University of Alb erta Edmonton, AB T6E 2E8, CA Russell Greiner r greiner@ualber t a.ca Dep artment of Computing Scienc e University of Alb erta Edmonton, AB T6E 2E8, CA Editor: Alexander Ihler Abstract W e in tro duce an efficien t message passing sc heme for solving Constrain t Satisfaction Prob- lems (CSPs), whic h uses sto chastic p erturbation of Belief Propagation (BP) and Survey Propagation (SP) messages to b ypass decimation and directly produce a single satisfying assignmen t. Our first CSP solver, called Perturb e d Belief Pr op agation , smo othly interpo- lates tw o w ell-kno wn inference pro cedures; it starts as BP and ends as a Gibbs sampler, whic h pro duces a single sample from the set of solutions. Moreo v er we apply a similar p erturbation sc heme to SP to pro duce another CSP solver, Perturb e d Survey Pr op agation . Exp erimen tal results on random and real-w orld CSPs sho w that P erturb ed BP is often more successful and at the same time tens to h undreds of times more efficient than standard BP guided decimation. P erturb ed BP also compares fa vorably with state-of-the-art SP-guided decimation, which has a computational complexity that generally scales exp onentially worse than our metho d (w.r.t. the cardinality of v ariable domains and constraints). F urthermore, our exp eriments with random satisfiabilit y and coloring problems demonstrate that P er- turb ed SP can outp erform SP-guided decimation, making it the b est incomplete random CSP-solv er in difficult regimes. Keyw ords: Constraint Satisfaction Problem, Message P assing, Belief Propagation, Survey Propagation, Gibbs Sampling, Decimation 1. In tro duction Probabilistic Graphical Mo dels (PGMs) pro vide a common ground for recent conv ergence of themes in computer science (artificial neural netw orks), statistical ph ysics of disordered systems (spin-glasses) and information theory (error correcting co des). In particular, mes- sage passing metho ds hav e b een successfully applied to obtain state-of-the-art solvers for Constrain t Satisfaction Problems (Mézard et al., 2002) The PGM formulation of a CSP defines a uniform distribution ov er the set of solutions, where each unsatisfying assignment has a zero probabilit y . In this framework, solving a CSP amoun ts to pro ducing a sample from this distribution. T o this end, usually an inference pro cedure estimates the marginal probabilities, which suggests an assignment to a subset of the most biased v ariables. This pro cess of sequen tially fixing a subset of v ariables, called 1 Ra v anbakhsh and Greiner de cimation , is rep eated un til all v ariables are fixed to pro duce a solution. Due to inaccuracy of the marginal estimates, this procedure gives an incomplete solver (Kautz et al., 2009), in the sense that the pro cedure’s failure is not a certificate of unsatisfiability . An alternative approac h is to use message passing to guide a searc h pro cedure that can back -track if a dead-end is reached ( e.g. , Kask et al., 2004; P arisi, 2003). Here using a branch and b ound tec hnique and relying on exact solv ers, one may also determine when a CSP is unsatisfiable. The most common inference procedure for this purp ose is Belief Propagation (Pearl, 1988). Ho w ev er, due to geometric prop erties of the set of s olutions (Krzak ala et al., 2007) as w ell as the complications from the decimation pro cedure (Co ja-Oghlan, 2011; Kro c et al., 2009), BP-guided decimation fails on difficult instances. The study of the change in the geometry of solutions has lead to Survey Propagation (Braunstein et al., 2002) which is a p o w erful message passing pro cedure that is slo wer than BP (p er iteration) but typically remains conv ergent, even in many situations when BP fails to con v erge. Using decimation, or other search schemes that are guided b y message passing, usually requires estimating marginals or partition functions, which is harder than pro ducing a single solution (V aliant, 1979). This pap er introduces a message passing scheme to eliminate this requiremen t, therefore also a v oiding the complications of applying decimation. Our alternativ e has adv antage o v er b oth BP- and SP-guided decimation when applied to solve CSPs. Here we consider BP and Gibbs Sampling (GS) updates as operators – Φ and Ψ resp ectiv ely – on a set of messages. W e then consider inference pro cedures that are conv ex com bination ( i.e. , γΨ + ( 1 − γ ) Φ ) of these t w o op erators. Our CSP solv er, Perturbed BP , starts at γ = 0 and ends at γ = 1 , smo othly changing from BP to GS, and finally producing a sample from the set of solutions. This change amounts to s tochastic biasing the BP messages tow ards the curren t estimate of marginals, where this random bias increases in eac h iteration. This pro cedure is often muc h more efficien t than BP-guided decimation (BP- dec) and sometimes succeeds where BP-dec fails. Our results on random CSPs (rCSPs) sho w that Perturbed BP is competitive with SP-guided decimation (SP-dec) in solving difficult random instances. Since SP can be interpreted as BP applied to an “auxiliary” PGM (Braunstein and Zecc hina, 2003), w e can apply the same p erturbation scheme to SP , which we call Perturbed SP . Note that this system, also, do es not p erform decimation and directly pro duce a solution (without using local searc h). Our experiments sho w that P erturb ed SP is often more successful than b oth SP-dec and Perturbed BP in finding satisfying assignments. Sto c hastic v ariations of BP ha ve b een previously proposed to perform inference in graphical mo dels ( e.g. , Ihler and Mcallester, 2009; No orshams and W ainwrigh t, 2013). Ho w ever, to our knowledge, P erturb ed BP is the first metho d to directly combine GS and BP up dates. In the follo wing, Section 1.1 in tro duces PGM form ulation of CSP using factor-graph notation. Section 1.2 reviews the BP equations and decimation pro cedure, then Section 1.3 casts GS as a message up date pro cedure. Section 2 introduces P erturb ed BP as a com- bination of GS and BP . Section 2.1 compares BP-dec and Perturbed BP on b enchmark CSP instances, sho wing that our metho d is often several folds faster and more successful in solving CSPs. Section 3 ov erviews the geometric prop erties of the set of solutions of rCSPs, then reviews first order Replica Symmetry Breaking Postulate and the resulting SP equations for CSP . Section 3.2 introduces Perturbed SP and Section 3.3 presents our exp erimen tal results for random satisfiabilit y and random coloring instances close to the unsatisfiabilit y threshold. Finally , Section 3.4 further discusses the b ehavior of decima- tion and p erturb ed BP in the light of a geometric picture of the set of solutions and the exp erimen tal results. 2 Per turbed Messa ge P assing f or CSP 1.1 F actor Graph Representation of CSP Let x = ( x 1 , x 2 , . . . , x N ) be a tuple of N discrete v ariables x i ∈ X i , where each X i is the domain of x i . Let I ⊆ N = { 1 , 2 , . . . , N } denote a subset of v ariable indices and x I = { x i | i ∈ I } b e the (sub)tuple of v ariables in x indexed by the subset I . Eac h constraint C I ( x I ) :  Q i ∈ I X i  → { 0 , 1 } maps an assignment to 1 iff that assignment satisfies that constrain t. Then the normalized pro duct of all constrain ts defines a uniform distribution o v er solutions: µ ( x ) , 1 Z Y I C I ( x I ) (1) where the partition function Z = P X Q I C I ( x I ) is equal to the num ber of solutions. 1 Notice that µ ( x ) is non-zero iff all of the constraints are satisfied – that is x is a solution. With slight abuse of notation we will use probability density and probabilit y distribution in terc hangeably . Example 1 ( q -COL:) Her e, x i ∈ X i = { 1 , . . . , q } is a q-ary variable for e ach i ∈ N , and we have M c onstr aints; e ach c onstr aint C i , j ( x i , x j ) = 1 − δ ( x i , x j ) dep ends only on two variables and is satisfie d iff the two variables have differ ent values (c olors). Her e δ ( x , x 0 ) is e qual to 1 if x = x 0 and 0 otherwise. This mo del can be conv enien tly represen ted as a bipartite graph, known as a factor gr aph (Kschisc hang et al., 2001), whic h includes t w o sets of no des: v ariable no des x i , and constrain t (or factor) nodes C I . A v ariable no de i (note that w e will often identify a v ariable “ x i ” with its index “ i ”) is connected to a constrain t node I if and only if i ∈ I . W e will use ∂ to denote the neighbors of a v ariable or constraint node in the factor graph – that is ∂I = { i | i ∈ I } (which is the set I ) and ∂i = { I | i ∈ I } . Finally we use ∆i to denote the Mark o v blanket of no de x i ( ∆i = { j ∈ ∂I | I ∈ ∂i , j 6 = i } ). The marginal of µ ( · ) for v ariable x i is defined as µ ( x i ) , X X N \ i µ ( x ) where the summation ab ov e is ov er all v ariables but x i . Belo w, we use b µ ( x i ) to denote an estimate of this marginal. Finally , w e use S to denote the (p ossibly empty) set of solutions S = { x ∈ X | µ ( x ) 6 = 0 } . Example 2 ( κ -SA T:) A l l variables ar e binary ( X i = { T rue , False } ) and e ach clause (c onstr aint C I ) dep ends on κ = | ∂I | variables. A clause evaluates to 0 only for a single assignment out of 2 κ p ossible assignment of variables (Gar ey and Johnson, 1979). Consider the fol lowing 3-SA T pr oblem over 3 variables with 5 clauses: SAT ( x ) = ( ¬ x 1 ∨ ¬ x 2 ∨ x 3 ) | {z } C 1 ∧ ( ¬ x 1 ∨ x 2 ∨ x 3 ) | {z } C 2 ∧ ( x 1 ∨ ¬ x 2 ∨ x 3 ) | {z } C 3 ∧ ( ¬ x 1 ∨ x 2 ∨ ¬ x 3 ) | {z } C 4 ∧ ( x 1 ∨ ¬ x 2 ∨ ¬ x 3 ) | {z } C 5 (2) The c onstr aint c orr esp onding to the first clause takes the value 1 , exc ept for x = { T rue , T rue , False } , in which c ase it is e qual to 0 . The set of solutions for this pr oblem is given by: S =  ( T rue , T rue , T rue ) , ( False , False , False ) , ( False , False , T rue )  . Figur e 1 shows the solutions as wel l as the c orr esp onding factor gr aph. 2 1. F or Eq 1 to remain v alid when the CSP is unsatisfiable, we define 0 0 , 0 . 2. In this simple case, we could combine all the constraints into a single constrain t ov er 3 v ariables and simplify the factor graph. Ho wev er, in general SA T, this cost-saving simplification is often not p ossible. 3 Ra v anbakhsh and Greiner (a) (b) Figure 1: (a) The set of all p ossible assignments to 3 v ariables. The solutions to the 3-SA T problem of Eq 2 are in white circles. (b) The factor-graph corresponding to the 3-SA T problem of Eq 2. Here eac h factor prohibits a single assignment. 1.2 Belief Propagation-guided Decimation Belief Propagation (Pearl, 1988) is a recursiv e up date pro cedure that sends a sequence of messages from v ariables to constraints ( ν i → I ) and vice-v ersa ( ν I → i ): ν i → I ( x i ) ∝ Y J ∈ ∂i \ I ν J → i ( x i ) (3) ν I → i ( x i ) ∝ X x I \ i ∈ X ∂I \ i C I ( x I ) Y j ∈ ∂I \ i ν j → I ( x j ) (4) where J ∈ ∂i \ I refers to all the factors connected to v ariable x i , except for factor C I . Similarly the summation in Eq 4 is o v er X ∂I \ i , means we are summing out all x j that are connected to C I ( i.e. , x j s . t . j ∈ I \ i ) except for x i . The me ssages are typically initialized to a uniform or a random distribution. This re- cursiv e up date of messages is usually p erformed until con v ergence – i.e. , until the maxim um c hange in the v alue of all messages, from one iteration to the next, is negligible ( i.e. , b elow some small  ). A t any p oint during the up dates, the estimated marginal probabilities are giv en by b µ ( x i ) ∝ Y J ∈ ∂i ν J → i ( x i ) (5) In a factor graph without lo ops, each BP message summarizes the effect of the (sub-tree that resides on the) sender-side on the receiving side. Example 3 Applying BP to the 3-SA T pr oblem of Eq 2 takes 20 iter ations to c onver ge (i.e., for the maximum change in the mar ginals to b e b elow  = 10 − 9 ). Her e the message, ν C 1 → 1 ( x 1 ) , fr om C1 to x 1 is: ν C 1 → 1 ( x 1 ) ∝ X x 2 , 3 C 1 ( x 1 , 2 , 3 ) ν 2 → C 1 ( x 2 ) ν 3 → C 1 ( x 3 ) Similarly, the message in the opp osite dir e ction, ν 1 → C 1 ( x 1 ) , is define d as ν 1 → C 1 ( x 1 ) ∝ ν C 2 → 1 ( x 1 ) ν C 3 → 1 ( x 1 ) ν C 4 → 1 ( x 1 ) ν C 5 → 1 ( x 1 ) 4 Per turbed Messa ge P assing f or CSP Her e BP gives us the fol lowing appr oximate mar ginals: b µ ( x 1 = T rue ) = b µ ( x 2 = T rue ) = . 319 and b µ ( x 3 = T rue ) = . 522 . F r om the set of solutions, we know that the c orr e ct mar ginals ar e b µ ( x 1 = T rue ) = b µ ( x 2 = T rue ) = 1/3 and b µ ( x 3 = T rue ) = 2/3 . The err or of BP is c ause d by influential lo ops in the factor-gr aph of Figur e 1(b). Her e the err or is r ather smal l; it c an b e arbitr arily lar ge in some instanc es or BP may not c onver ge at al l. The time complexity of BP updates of Eq 3 and Eq 4, for eac h of the messages exchanged b et w een i and I , are O ( | ∂i | | X i | ) and O ( | X I | ) respectively . W e ma y reduce the time complexity of BP by synchronously up dating all the me ssages ν i → I ∀ I ∈ ∂i that lea ve no de i . F or this, w e first calculate the b eliefs b µ ( x i ) using Eq 5 and pro duce each ν i → I using ν i → I ( x i ) ∝ b µ ( x i ) ν I → i ( x i ) . (6) Note than we can substitute Eq 4 in to Eq 3 and Eq 5 and only keep v ariable-to-factor messages. After this substitution, BP can b e viewed as a fixed-p oint iteration pro cedure that rep eatedly applies the op erator Φ ( { ν i → I } ) , { Φ i → I ( { ν j → J } j ∈ ∆i , J ∈ ∂i \ I } ) } i , I ∈ ∂i to the set of messages in hop e of reaching a fixed p oint: ν i → I ( x i ) ∝ Y J ∈ ∂i \ I X X ∂J \ i C J ( x J ) Y j ∈ ∂J \ i ν j → J ( x j ) , Φ i → I ( { ν j → J } j ∈ ∆i , J ∈ ∂i \ I )( x i ) (7) Also Eq 5 b ecomes b µ ( x i ) ∝ Y I ∈ ∂i X X ∂I \ i C I ( x I ) Y j ∈ ∂I \ i ν j → I ( x j ) (8) where Φ i → I denotes individual message up date operators. W e let op erator Φ ( . ) denote the set of these Φ i → I op erators. 1.2.1 Decima tion The decimation pro cedure can emplo y BP (or SP) to solve a CSP . W e refer to the cor- resp onding metho d as BP-dec (or SP-dec). After running the inference pro cedure and obtaining b µ ( x i ) , ∀ i , the decimation pro cedure uses a heuristic approach to select the most biased v ariables (or just a random subset) and fixes these v ariables to their most biased v alues (or a random b x i ∼ b µ ( x i ) ). If it selects a fraction ρ of remaining v ariables to fix after eac h conv ergence, this m ultiplies an additional log 1 ρ ( N ) to the linear (in N ) cost 3 for each iteration of BP (or SP). The following algorithm 1 summarizes BP-dec with a particular sc heduling of up dates: The condition of line 9 is satisfied iff the pro duct of incoming messages to no de i is 0 for all x i ∈ X i . This means that neighboring constraints ha ve strict disagreement ab out the v alue of x i and the decimation has found a con tradiction. This con tradiction can happ en because, either (I) there is no solution for the reduced problem ev en if the original problem had a solution, or (I I) the reduced problem has a solution but the BP messages are inaccurate. Example 4 T o apply BP-de c to pr evious example, we first c alculate BP mar ginals, as shown in the example ab ove. Her e b µ ( x 1 ) and b µ ( x 2 ) have the highest bias. By fixing the value of x 1 to False , the 3. Assuming the num b er of edges in the factor graph are in the order of N . In general, using synchronous up date of Eq 6 and assuming a constant factor cardinality , | ∂I | , the cost of eac h iteration is O ( E ) , where E is the num b er of edges in the factor-graph. 5 Ra v anbakhsh and Greiner Algorithm 1: Belief Propagation-guided Decimation (BP-dec) input : factor-graph of a CSP output : a satisfying assignment x ∗ if an assignmen t was found. unsa tisfied otherwise 1 initialize the messages 2 e N ← N (set of all v ariable indices) 3 while e N is not empty do // decimation loop 4 5 rep eat // BP loop 6 7 foreac h i ∈ e N do 8 calculate messages { ν I → i } I ∈ ∂i using Eq 4 9 if { ν I → i } I ∈ ∂i ar e c ontr adictory then return : unsa tisfied 10 calculate marginal b µ ( x i ) using Eq 5 11 calculate messages { ν i → I } I ∈ ∂i using Eq 3 or Eq 6 12 un til c onver genc e 13 select B ⊆ e N using { b µ ( x i ) } i ∈ e N 14 fix x ∗ j ← arg x j max b µ ( x j ) ∀ j ∈ B 15 reduce the constrain ts { C I } I ∈ ∂j for every j ∈ B return : x ∗ = ( x ∗ 1 , . . . , x ∗ N ) SA T pr oblem of Eq 2 c ol lapses to: SAT ( x { 2 , 3 } ) | x 1 = False = ( ¬ x 2 ∨ x 3 ) ∧ ( ¬ x 2 ∨ ¬ x 3 ) (9) BP-de c applies BP again to this r e duc e d pr oblem, which give b µ ( x 2 = T rue ) = . 14 (note her e that µ ( x 2 = T rue ) = 0 ) and b µ ( x 3 = T rue ) = 1/2 . By fixing x 2 to False , another r ound of de cimation yields a solution x ∗ = ( False , False , T rue ) . 1.3 Gibbs Sampling as Message Up date Gibbs Sampling (GS) is a Mark ov Chain Monte Carlo (MCMC) inference pro cedure (An- drieu et al., 2003) that can pro duce a set of samples b x [ 1 ] , . . . , b x [ L ] from a given PGM. W e can then reco v er the marginal probabilities, as empirical exp ectations: b µ L ( x i ) ∝ 1 L L X n = 1 δ ( b x [ n ] i , x i ) (10) Our algorithm only considers a single particle b x = b x [ 1 ] . GS starts from a random initial state b x ( t = 0 ) and at eac h time-step t , up dates each b x i b y sampling from: b x ( t ) i ∼ µ ( x i ) ∝ Y I ∈ ∂i C I ( x i , b x ( t − 1 ) ∂I \ i ) (11) 6 Per turbed Messa ge P assing f or CSP If the Marko v chain satisfies certain basic prop erties (Rob ert and Casella, 2005), x ( ∞ ) i is guaranteed to be an unbiased sample from µ ( x i ) and therefore our marginal estimate, b µ L ( x i ) , b ecomes exact as L → ∞ . In order to interpolate betw een BP and GS, we establish a corresp ondence b etw een a particle in GS and a set of v ariable-to-factor messages – i.e. , b x ⇔ { ν i → I ( . ) } i , I ∈ ∂i . Here all the messages lea ving v ariable x i are equal to a δ -function defined based on b x i : ν i → I ( x i ) = δ ( x i , b x i ) ∀ I ∈ ∂i (12) W e define the random GS op erator Ψ = { Ψ i } i and rewrite the GS up date of Eq 11 as ν i → I ( x i ) , Ψ i ( { ν j → J ( x j ) } j ∈ ∆i , J ∈ ∂i )( x i ) = δ ( b x i , x i ) (13) where b x i is sampled from b x i ∼ b µ ( x i ) ∝ Y J ∈ ∂i C I ( x i , b x ∂I \ i ) ∝ Y I ∈ ∂i X X ∂I \ i C I ( x I ) Y j ∈ ∂I \ i ν j → I ( x j ) (14) Note that Eq 14 is identical to BP estimate of the marginal Eq 8. This equality is a consequence of the wa y w e hav e defined messages in the GS up date and allows us to com bine BP and GS up dates in the following section. 2. P erturb ed Belief Propagation Here we introduce an alternativ e to decimation that do es not require rep eated application of inference. The basic idea is to use a linear combination of BP and GS op erators (Eq 7 and Eq 13) to up date the messages: Γ ( { ν i → I } ) , γ Ψ ( { ν i → I } ) + ( 1 − γ ) Φ ( { ν i → I } ) (15) The P erturb ed BP op erator Γ = { Γ i → I } i , I ∈ ∂i up dates eac h message by calculating the outgoing message according to BP and GS op erators and linearly combines them to get the final mess age. During T iterations of Perturbed BP , the parameter γ is gradually and linearly changed from 0 to w ards 1 . Algorithm 2 b elow summarizes this pro cedure. In step 7, if the pro duct of incoming messages is 0 for all x i ∈ X i for some i , different neigh b oring constraints ha v e strict disagreemen t ab out x i ; therefore this run of Perturbed BP will not b e able to satisfy this CSP . Since the pro cedure is inheren tly sto chastic, if the CSP is satisfiable, re-application of the same pro cedure to the problem ma y av oid this sp ecific contradiction. 2.1 Exp erimen tal Results on Benc hmark CSP This section compares the performance of BP-dec and Perturbed BP on benchmark CSPs. W e considered CSP instances from XCSP rep ository (Roussel and Lecoutre, 2009; Lecoutre, 2013), without global constraints or complex domains. 4 4. All instances with intensiv e constraints ( i.e. , functional form) were con verted in to extensive format for explicit represen tation using dense factors. W e further remov ed instances containing constraints with more that 10 6 en teries in their tabular form. W e also discarded instances that collectively had more than 10 8 en teries in the dense tabular form of their constrain ts. Since our implementation represen ts all factors in a dense tabular form, we had to remov e man y instances b ecause of their large factor size. W e an ticipate that Perturbed BP and BP-dec could probably solve many of these instances using a sparse represen tation. 7 Ra v anbakhsh and Greiner Algorithm 2: Perturbed Belief Propagation input : factor graph of a CSP , n um b er of iterations T output : a satisfying assignment x ∗ if an assignmen t was found. unsa tisfied otherwise 1 initialize the messages 2 γ ← 0 3 e N ← N (set of all v ariable indices) 4 for t = 1 to T do 5 foreac h variable x i do 6 calculate ν I → i using Eq 4 ∀ I ∈ ∂i 7 if { ν I → i } I ∈ ∂i ar e c ontr adictory then return : unsa tisfied 8 calculate marginals b µ ( x i ) using Eq 14 9 calculate BP messages ν i → I using Eq 3 or Eq 6 ∀ I ∈ ∂i . 10 sample b x i ∼ b µ ( x i ) 11 com bine BP and Gibbs sampling messages: ν i → I ← γ ν i → I + ( 1 − γ ) δ ( x i , b x i ) 12 γ ← γ + 1 T − 1 return : x ∗ = { x ∗ 1 , . . . , x ∗ N } W e used a conv ergence threshold of  = . 001 for BP and terminated if the threshold w as not reached after T = 10 × 2 10 = 10 , 24 0 iterations. T o p erform decimation, w e sort the v ariables according to their bias and fix ρ fraction of the most biased v ariables in each iteration of decimation. This fraction, ρ , was initially set to 100 %, and it w as divided b y 2 eac h time BP-dec failed on the same instance. BP-dec was rep eatedly applied using the reduced ρ , at most 10 times, unless a solution was reached (that is ρ = . 1 % at final attempt). F or P erturbed BP , we set T = 10 at the starting attempt, which was increased by a factor of 2 in case of failure. This w as rep eated at most 10 times, which means Perturbed BP used T = 10 , 240 at its final attempt. Note that P erturb ed BP at most uses the same n um b er of iterations as the maxim um iterations p er single iteration of decimation in BP- dec. Figure 2(a,b) compares the time and iterations of BP-dec and Perturbed BP for success- ful attempts where b oth metho ds satisfied an instance. The result for individual problem- sets is rep orted in the app endix. Empirically , we found that P erturb ed BP b oth solv ed (slightly) more instances than BP-dec (284 vs 253), and w as (h undreds of times) more efficien t: while Perturbed BP required only 133 iterations on av erage, BP-dec required an av erage of 41,284 iterations for successful instances. W e also ran BP-dec on all the b enchmarks with maximum n um b er of iterations set to T = 1000 and T = 100 iterations. This reduced the n um ber of satisfied instances to 249 for T = 1000 and 247 for T = 100 , but also reduced the av erage num ber of iterations to 1570 and 562 resp ectively , whic h are still several folds more exp ensive than Perturbed BP . Figure 2(c-f ) compare the time and iterations used by BP-dec in these settings with that of Perturbed BP , when b oth metho ds found a satisfying as signmen t. See the app endix for a more detailed rep ort on these results. 8 Per turbed Messa ge P assing f or CSP 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 1 0 5 1 0 6 1 0 7 BP-dec (iters) 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 Perturbed BP (iters) (a) iterations 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 1 0 5 BP-dec (time) 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 Perturbed BP (time) (b) time (seconds) 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 1 0 5 BP-dec (iters) 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 Perturbed BP (iters) (c) iterations for T = 1000 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 BP-dec (time) 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 Perturbed BP (time) (d) time (seconds) for T = 1000 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 BP-dec (iters) 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4 Perturbed BP (iters) (e) iterations for T = 100 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 BP-dec (time) 1 0 - 4 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 Perturbed BP (time) (f ) time (seconds) for T = 100 Figure 2: Comparison of time and num ber of iterations used by BP-dec and Perturbed BP in b enc hmark instances where both metho ds found satisfying assignmen ts. (a,b) Maximum n umber of BP iterations p er iteration of decimation is T = 10 , 240 , equal to the maximum iterations used by P erturb ed BP . (c,d) Maximum num b er of iterations for BP in BP-dec is reduced to T = 1000 . (e,f ) Maxim um num b er of iterations for BP in BP-dec is further reduced to T = 100 . 9 Ra v anbakhsh and Greiner 3. Critical Phenomena in Random CSPs Random CSP (rCSP) instances ha v e been extensively us ed in order to study the prop er- ties of combinatorial problems (Mitchell et al., 1992; Ac hioptas and Sorkin, 2000; Krzak ala et al., 2007) as w ell as in analysis and design of algorithms ( e.g. , Selman et al., 1994; Mézard et al., 2002). Random CSPs are closely related to spin-glasses in statistical physics (Kirkpatric k and Selman, 1994; F u and Anderson, 1986). This connection follows from the fact that the Hamiltonian of these spin-glass systems resembles the ob jective functions in man y com binatorial problems, whic h decomp ose to pairwise (or higher order) interactions, allo wing for a graphical representation in the form of a PGM. Here message passing meth- o ds, suc h as b elief propagation (BP) and survey propagation (SP), pro vide consistency conditions on lo cally tree-like neigh b orhoo ds of the graph. The analogy b et w een a ph ysical system and computational problem extends to their critical b eha vior where computation relates to dynamics (Ricci-T ersenghi, 2010). In com- puter science, this critical b ehavior is related to the time-complexity of algorithms employ ed to solve such problems, while in spin-glass theory this translates to dynamics of glassy state, and exp onential relaxation times (Mézard et al., 1987). In fact, this connection has b een used to attempt to pro v e the conjecture that P is not equal to NP (Deolalik ar, 2010). Studies of rCSP , as a critical phenomena, fo cus on the geometry of the solution space as a function of the problem’s difficulty , where rigorous ( e.g. , Ac hlioptas and Co ja-Oghlan, 2008; Co cco et al., 2003) and non-rigorous ( e.g. , Cavit y metho d of Mézard and P arisi (2001) and Mézard and Parisi (2003)) analyses hav e confirmed the same geometric picture. When w orking with large random instances, a scalar α asso ciated with a problem instance, a.k.a. con trol parameter – for example, the clause to v ariable ratio in SA T – can characterize that instance’s difficulty ( i.e. , larger con trol parameter corresp onds to a more difficult instance) and in many situations it characterizes a sharp transition from satisfiabilit y to unsatisfiabilit y (Cheeseman et al., 1991). Example 5 (Random κ -SA T) R andom κ -SA T instanc e with N variables and M = αN c onstr aints ar e gener ate d by sele cting κ variables at r andom for e ach c onstr aint. Each c onstr aint is set to zer o (i.e., unsatisfie d) for a single r andom assignment (out of 2 κ ). Her e α is the c ontr ol p ar ameter. Example 6 (Random q -COL) The c ontr ol p ar ameter for a r andom q -COL instanc es with N vari- ables and M c onstr aints is its aver age de gr e e α = 2M N . W e c onsider Er dős-R ény r andom gr aphs and gener ate a r andom instanc e by se quential ly sele cting two distinct variables out of N at r andom to gener ate e ach of M e dges. F or lar ge N , this is e quivalent to sele cting e ach p ossible factor with a fixe d pr ob ability, which me ans the no des have Poisson de gr e e distribution P ( | ∂i | = d ) ∝ e − α α d . While there are tight b ounds for some problems ( e.g. , Ac hlioptas et al., 2005), finding the exact lo cation of this transition for different CSPs is still an op en problem. Besides transition to unsatisfiability , these analyses has revealed several other (phase) transitions (Krzak ala et al., 2007). Figure 3(a)-(c) sho ws how the geometry of the set of solutions c hanges by increasing the con trol parameter. Here w e en umerate v arious phases of the problem for increasing v alues of the con trol parameter: (a) In the so-called R eplic a Symmetric (RS) phase, the symmetries of the set of solutions (a.k.a. ground states) reflect the trivial symmetries of problem w.r.t. v ariable domains. F or example, for q -COL the set of solutions is symmetric w.r.t. sw apping all red and blue assignment. In this regime, the set of solutions form a giant cluster ( i.e. , a set of neigh b oring solutions), where tw o solutions are considered neighbors when their Ham- ming distance is one (A c hlioptas and Co ja-Oghlan, 2008) or non-divergen t with num b er of v ariables (Mézard and P arisi, 2003). Lo cal search metho ds ( e.g. , Selman et al., 1994) and BP-dec can often efficiently solv e random CSPs that b elong to this phase. 10 Per turbed Messa ge P assing f or CSP (a) Replica Symmetric (b) clustering (c) condensation Figure 3: A 2-dimensional sc hematic view of ho w the set of solutions of CSP v aries as we increase the con trol parameter α from (a) replica symmetric phase to (b) clustering phase to (c) condensation phase. Here small circles represent solutions and the bigger circles represent clusters of solutions. Note that this view is v ery simplistic in many w a ys; for example, the total num b er of solutions and the size of clusters should generally decrease from (a) to (c). (b) In clustering or dynamic al transition (1dRSB 5 ), the set of solutions decomp oses into an exp onen tial n um b er of distan t clusters. Here tw o clusters are distant if the Hamming distance betw een their respective mem b ers is divergen t ( e.g. , linear) in the n umber of v ariables. (c) In the c ondensation phase transition (1sRSB 6 ), the set of solutions condenses in to a few dominan t clusters. Dominant clusters ha v e roughly the same num ber of solutions and they collectiv ely contain almost all of the solutions. While SP can b e used even within the condensation phase, BP usually fails to conv erge in this regime. Ho wev er each cluster of solutions in the clustering and condensation phase is a v alid fixed-point of BP , whic h is called a “quasi-solution” of BP . (d) A rigidity transition (not included in Figure 3) iden tifies a phase in which a finite p ortion of v ariables are fixed within dominant clusters. This transition triggers an exp onential decrease in the total num b er of solutions, whic h leads to (e) unsatisfiability transition. 7 This rough picture summarizes first order Replica Symmetry Breaking’s (1RSB) basic assumptions (Mézard and Montanari, 2009). F rom a geometric persp ective, the intuitiv e idea b ehind Perturbed BP , is to p erturb the messages tow ards a solution. Ho w ev er, in order to achiev e this, we need to initialize the messages to a proper neighborho o d of a solution. Since these neighborho o ds are not initially known, we resort to sto chastic p erturbation of messages to make lo cal marginals more biased tow ards a subspace of solutions. This contin uous p erturbation of all mes- sages is performed in a wa y that allo ws eac h BP message to re-adjust itself to the other p erturbations, more and more fo cusing on a random subset of solutions. 3.1 1RSB P ostulate and Survey Propagation Large random graphs are lo cally tree-lik e, which means the length of short lo ops are typ- ically in the order of log ( N ) (Janson et al., 2001). This ensures that, in the absence of long-range correlations, BP is asymptotically exact, as the set of messages incoming to eac h node or factor are almost indep endent. Although BP messages remain uncorrelated un til the condensation transition (Krzak ala et al., 2007), the BP equations do not com- pletely characterize the set of solutions after the clustering transition. This inadequacy 5. 1st order dynamical RSB. The term Replica Symmetry Breaking (RSB) originates from the tec hnique – i.e. , Replica tric k (Mézard et al. 1987) – that was first used to analyze this setting. According to RSB, the trivial symmetries of the problem do not c haracterize the clusters of solution. 6. 1st order static RSB. 7. In some problems, the rigidit y transition o ccurs b efore condensation transition. 11 Ra v anbakhsh and Greiner is indicated by the existence of a set of several v alid fixed p oin ts (rather than a unique fixed-p oin t) for BP , eac h of which corresp onds to a quasi-solution. F or a b etter intuition, consider the carto ons of Figures 3(b) and (c). During the clustering phase (b), x i and x j (corresp onding to the x and y axes) are not highly correlated, but they b ecome correlated during and after condensation (c). This correlation betw een v ariables that are far apart in the PGM results in correlation b et w een the BP messages. This violates BP’s assumption that messages are uncorrelated, whic h results in BP’s failure in this regime. 1RSB’s approac h to incorporating this clustering of solutions into the equilibrium con- ditions is to define a new Gibbs measure o ver clusters. Let y ⊂ S denote a cluster of solutions and Y be the set of all suc h clusters. The idea is to treat Y the same as we treated X , by defining a distribution µ ( y ) ∝ | y | m ∀ y ∈ Y (16) where m ∈ [ 0 , 1 ] , called the P arisi parameter (Mézard et al., 1987), specifies how each cluster’s weigh t dep ends on its size. This implicitly defines a distribution ov er X µ ( x ) ∝ X y 3 x µ ( y ) (17) N.b., m = 1 corresp onds to the original distribution (Eq 1). Example 7 Going b ack to our simple 3-SA T example, y ( 1 ) = { ( T rue , T rue , T rue ) } and y ( 2 ) = { ( False , False , False ) , ( False , False , T rue ) } ar e two clusters of solutions. Using m = 1 , we have µ ( {{ T rue , T rue , T rue }} ) = 1/3 and µ ( {{ False , False , False } , { False , False , T rue }} ) = 2/3 . This distribu- tion over clusters r epr o duc es the distribution over solutions – i.e., µ ( x ) = 1/3 ∀ x ∈ S . On the other hand, using m = 0 , pr o duc es a uniform distribution over clusters, but it do es not give us a uniform distribution over the solutions. This meta-construction for µ ( y ) can be represen ted using an auxiliary PGM. One ma y use BP to find marginals o v er this PGM; here BP messages are distributions ov er all BP messages in the original PGM, as each cluster is a fixed-p oin t for BP . This requirement to represen t a distribution o v er distributions makes 1RSB practically intractable. In general, eac h original BP message is a distribution o ver X i and it is difficult to define a distribution o v er this infinite set. Ho wev er this simplifies if the original BP messages can hav e limited v alues. F ortunately if w e apply max-pro duct BP to solve a CSP , instead of sum-pro duct BP (of Eqs 3 and 4), the messages can hav e a finite set of v alues. Max-Pro duct BP: Our previous form ulation of CSP w as using sum-pro duct BP . In gen- eral, max-product BP is used to find the Maxim um a Posteriori (MAP) assignment in a PGM, which is a single assignment with the highest probability . In our PGM, the MAP assignmen t is a solution for the CSP . The max-pro duct up date equations are η i → I ( x i ) = Q J ∈ ∂i \ I η J → i ( x i ) = Λ i → I ( { η J → i } J ∈ ∂i \ I )( x i ) (18) η I → i ( x i ) = max X ∂I \ i C I ( x I ) Q j ∈ ∂I \ i η j → I ( x j ) = Λ I → i ( { η j → I } j ∈ ∂I \ i )( x i ) (19) b µ ( x i ) = Q J ∈ ∂i η J → i ( x i ) = Λ i ( { η J → i } J ∈ ∂i )( x i ) (20) where Λ = { Λ i → I , Λ I → i } i , I ∈ ∂I is the max-pro duct BP op erator and Λ i represen ts the marginal estimate as a function of messages. Note that here messages and marginals are not distributions. W e initialize ν i → I ( x i ) ∈ { 0 , 1 } , ∀ I , i ∈ ∂I , x i ∈ X i . Because of the 12 Per turbed Messa ge P assing f or CSP w a y constraints and up date equations are defined, at any p oint during the up dates we ha v e ν i → I ( x i ) ∈ { 0 , 1 } . This is also true for b µ ( x i ) . Here any of ν i → I ( x i ) = 1 , ν I → i ( x i ) = 1 or b µ ( x i ) = 1 , shows that v alue x i is allow ed according to a message or marginal, while 0 forbids that v alue. Note that b µ ( x i ) = 0 ∀ x i ∈ X i iff no solution w as found, b ecause the incoming messages were contradictory . The non-trivial fixed-p oints of max-pro duct BP define quasi-solutions in 1RSB phase, and therefore define clusters y . Example 8 If we initialize al l messages to 1 for our simple 3-SA T example, the final mar ginals over al l the variables ar e e qual to 1, al lowing al l assignments for al l variables. However b eside this trivial fixe d-p oint, ther e ar e other fixe d p oints that c orr esp ond to two clusters of solutions. F or example, c onsidering the cluster { ( False , False , False ) , ( False , False , T rue ) } , the fol lowing { η i → I } (and their c orr esp onding { η I → i } define a fixe d-p oint for max-pr o duct BP: η 1 → I ( T rue ) = b µ 1 ( T rue ) = 0 η 1 → I ( False ) = b µ 1 ( False ) = 1 ∀ I ∈ ∂1 η 2 → I ( T rue ) = b µ 2 ( T rue ) = 0 η 2 → I ( False ) = b µ 2 ( False ) = 1 ∀ I ∈ ∂2 η 3 → I ( T rue ) = b µ 3 ( T rue ) = 1 η 3 → I ( False ) = b µ 3 ( False ) = 1 ∀ I ∈ ∂3 Her e the messages indic ate the al lowe d assignments within this p articular cluster of solutions. 3.1.1 Sur vey Pr op aga tion Here w e define the 1RSB up date equations ov er max-pro duct BP messages. W e skip the explicit construction of the auxiliary PGM that results in SP up date equations, and confine this section to the intuition offered b y SP messages. F or the construction of the auxiliary- PGM see (Braunstein and Zecc hina, 2003) and (Mézard and Montanari, 2009). See (Manev a et al., 2007) for a different p ersp ective on the relation of BP and SP for the satisfiability problem and (Kro c et al., 2007) for an exp erimental study of SP applied to SA T. Let Y i = 2 | X i | b e the p o wer-set 8 of X i . Eac h max-pro duct BP message can b e seen as a subset of X i that contains the allow ed states. Therefore Y i as its p o w er-set contains all p ossible max-product BP messages. Each message ν i → I : Y i → [ 0 , 1 ] in the auxiliary PGM defines a distribution o v er original max-pro duct BP messages. Example 9 (3-COL) X i = { 1 , 2 , 3 } is the set of c olors and Y i = {{} , { 1 } , { 2 } , { 3 } , { 1 , 2 } , { 2 , 3 } , { 1 , 3 } , { 1 , 2 , 3 }} . Her e y i = {} c orr esp onds to the c ase wher e none of the c olors ar e al lowe d. Applying sum-pro duct BP to our auxiliary PGM gives en tropic SP( m ) up dates as: ν i → I ( y i ) ∝ | y i | m X { η J → i } J ∈ ∂i \ I δ ( y i , Λ i → I ( { η J → i } J ∈ ∂i \ I )) Y J ∈ ∂i \ I ν J → i ( η J → i ) (21) ν I → i ( y i ) ∝ | y i | m X { η j → I } j ∈ ∂I \ i δ ( y i , Λ I → i ( { η j → I } j ∈ ∂I \ i )) Y j ∈ ∂I \ i ν j → I ( η j → I ) (22) ν i → I ( {} ) := ν I → i ( {} ) := 0 ∀ i , I ∈ ∂i (23) where the summations are ov er all combinations of max-product BP messages. Here the δ -function ensures that only the set of incoming messages that satisfy the original BP equations make contributions. Since we only care ab out the v alid assignments and y i = {} forbids all assignmen ts, we ignore its con tribution (Eq 23). 8. The pow er-set of X is the set of all subsets of X , including {} and X itself. 13 Ra v anbakhsh and Greiner Example 10 (3-SA T) Consider the SP message ν 1 → C 1 ( y 1 ) in the factor gr aph of Figur e 1b. Her e the summation in Eq 21 is over al l p ossible c ombinations of inc oming max-pr o duct BP messages η C 2 → 1 , . . . , η C 5 → 1 . Sinc e e ach of these messages c an assume one of the thr e e valid values – e.g., η C 2 → 1 ( x 1 ) ∈ { { T rue } , { False } , { T rue , False } } – for e ach p articular assignment of y 1 , a total of |{{ T rue } , { False } , { T rue , False }}| | ∂1 \ C 1 | = 3 4 p ossible c ombinations ar e enumer ate d in the summations of Eq 21. However only the c ombinations that form a valid max-pr o duct message up date have non- zer o c ontribution in c alculating ν 1 → C 1 ( y 1 ) . These ar e b asic al ly the messages that app e ar in a max- pr o duct fixe d p oint as discusse d in Example 8. Eac h of original messages corresp onds to a different sub-set of clusters and m (from Eq 16) controls the effect of each cluster’s size on its contribution. A t an y p oint, w e can use these messages to estimate the marginals of b µ ( y ) defined in Eq 16 b µ ( y i ) ∝ | y i | m X { η J → i } J ∈ ∂i δ ( y i , Λ i ( { η J → i } J ∈ ∂i ) ) Y J ∈ ∂i ν J → i ( η J → i ) (24) This also implies a distribution ov er the original domain, which we slightly abuse no- tation to denote by: b µ ( x i ) ∝ X y i 3 x i b µ ( y i ) (25) The term SP usually refers to SP( 0 ) – i.e. , m = 0 – where all clusters, regardless of their size, contribute the same amount to µ ( y ) . Now that we can obtain an estimate of marginals, we can employ this pro cedure within a decimation pro cess to incremen tally fix some v ariables. Here either b µ ( x i ) or b µ ( y i ) can b e used b y the decimation pro cedure to fix the most biased v ariables. In the former case, a v ariable y i is fixed to y ∗ i = { x ∗ i } when x ∗ i = arg x i max b µ ( x i ) . In the latter case, y ∗ i = arg y i max b µ ( y i ) . Here we use SP-dec(S) to refer to the former pro cedure (that uses b µ ( x i ) to fix v ariables to a single v alue) and use SP-dec(C) to refer to the later case (in which v ariables are fixed to a cluster of assignmen ts). The original decimation proced ure for κ -SA T (Braunstein et al., 2002) corresp onds to SP-dec(S). SP-dec(C) for CSP with Bo olean v ariables is only slightly different, as SP-dec(C) can choose to fix a cluster to y i = { T rue , False } in addition to the options of y i = { T rue } and y i = { False } , av ailable to SP-dec(S). How ever, for larger domains ( e.g. , q -COL), SP-dec(C) has a clear adv antage. F or example in 3 -COL, SP-dec(C) may choose to fix a cluster to y i = { 1 , 2 } while SP-dec(S) can only choose b etw een y i ∈ {{ 1 } , { 2 } , { 3 }} . This significant difference is also reflected in their comparative success-rate on q -COL. 9 (See T able 1 in Section 3.3.) During the decimation pro cess, usually after fixing a subset of v ariables, the SP marginals b µ ( x i ) b ecome uniform, indicating that clusters of solutions hav e no preference ov er partic- ular assignments of the remaining v ariables. The same happ ens when we apply SP to random instances in RS phase. At this p oint (a.k.a. paramagnetic phase), a lo cal search metho d or BP-dec can often efficiently find an assignmen t to the v ariables that are not yet fixed by decimation. Note that both SP-dec(C) and SP-dec(S) switch to local searc h as so on as all b µ ( x i ) b ecome close to uniform. The computational complexity of each SP up date of Eq 22 is O ( 2 | X i | − 1 ) | ∂I | as for each particular v alue y i , SP needs to consider every com bination of incoming messages, eac h of whic h can tak e 2 | X i | v alues (minus the empty set). Similarly , using a naive approach the cost of up date of Eq 21 is O ( 2 | X i | − 1 ) | ∂i | . Ho wev er b y considering incoming messages one 9. Previous applications of SP-dec to q -COL b y Braunstein et al. (2003) used a heuristic for decimation that is similar SP-dec (C). 14 Per turbed Messa ge P assing f or CSP at a time, we can perform the same exact up date in O ( | ∂i | 2 2 | X i | ) . In comparison to the cost of BP up dates, we see that SP up dates are substantially more expensive for large | X i | and | ∂I | . 10 3.2 P erturb ed Surv ey Propagation The p erturbation scheme that we use for SP is similar to what we did for BP . Let Φ i → I ( { ν j → J } j ∈ ∆i , ( J ∈ ∂i ) \ I ) ) denote the up date operator for the message from v ariable y i to factor C I . This op erator is obtained by substituting Eq 22 into Eq 21 to get a single SP up date equation. Let Φ ( { ν i → I } i , I ∈ ∂i ) denote the aggregate SP op erator, which applies Φ i → I to up date each individual message. W e p erform Gibbs sampling from the “original” domain X using the implicit marginal of Eq 25. W e denote this random op erator by Ψ = { Ψ i } i : ν i → I ( y i ) = Ψ i ( { ν j → J } j ∈ ∆i , J ∈ ∂i ) , δ ( y i , { b x i } ) where b x i ∼ b µ ( x i ) (26) where the second argumen t of the δ -function is a singleton set, containing a sample from the estimate of marginal. No w, define the P erturb ed SP op erator as the conv ex com bination of SP and either of the GS op erator ab ov e: Γ ( { ν i → I } ) , γ Ψ ( { ν i → I } ) + ( 1 − γ ) Φ ( { ν i → I } ) (27) Similar to p erturb ed BP , d uring iterations of Perturbed SP , γ is gradually increased from 0 to 1 . If p erturb ed SP reaches the final iteration, the samples from the implicit marginals represen t a satisfying assignmen t. The adv an tage of this sc heme to SP-dec is that p erturb ed SP do es not require any further local searc h. In fact w e may apply Γ to CSP instances in the RS phase as w ell, where the solutions form a single giant cluster. In con trast, applying SP-dec, to these instances simply inv okes the lo cal searc h metho d. T o demonstrate this, we applied Perturbed SP(S) to b enchmark CSP instances of T a- ble 2 in whic h the maximum num b er of elements in the factor was less than 10 . Here P erturb ed SP(S) solv ed 80 instances out of 202 cases, while P erturb ed BP solv ed 78 in- stances. 3.3 Exp erimen ts on random CSP W e implemented all the metho ds ab ov e for general factored CSP using the lib dai co de base (Mo oij, 2010). T o our knowledge this is the first general implementation of SP and SP-dec. Previous applications of SP-dec to κ -SA T and q -COL (Braunstein et al., 2003; Mulet et al., 2002; Braunstein et al., 2002) were specifically tailored to just one of those problems. Here we report the results on κ -SA T for κ ∈ { 3 , 4 } and q -COL for q ∈ { 3 , 4 , 9 } . W e used the pro cedure discussed in the examples of Section 3 to produce 100 random instances with N = 5 , 000 v ariables for eac h control parameter α . W e report the probabilit y of finding a satisfying assignment for differen t metho ds ( i.e. , the p ortion of 100 instances that were satisfied by each method). F or coloring instances, to help decimation, w e break the initial symmetry of the problem b y fixing a single v ariable to an arbitrary v alue. F or BP-dec and SP-dec, w e use a conv ergence threshold of  = . 001 and fix ρ = 1 % of v ariables p er iteration of decimation. P erturb ed BP and Perturbed SP use T = 1000 10. Note that our represen tation of distributions is ov er-complete – that is we are not using the fact that the distributions sum to one. How ever ev en in their more compact forms, for general CSPs, the cost of eac h SP up date remains exp onentially larger than that of BP (in | X i | , | ∂I | ). How ev er if the factors are sparse and hav e high order, b oth BP and SP allow more efficien t updates. 15 Ra v anbakhsh and Greiner Figure 4: (first ro w) Success-rate of different metho ds for 3-COL and 3-SA T for v arious control parameters. (second row) The a verage num b er of v ariables (out of N = 5000 ) that are fixed using SP-dec (C) and (S) before calling local searc h, a v eraged o ver 100 instances. (third ro w) The a v erage amount of time (in seconds) used by the successful setting of eac h metho d to find a satisfying solution. F or SP-dec(C) and (S) this includes the time used b y lo cal search. (fourth row) The n um b er of iterations used by differen t metho ds at different control parameters, when the metho d w as successful at finding a solution. The num b er of iterations for each of 100 random instances is rounded to the closest p o w er of 2. This do es not include the iterations used b y lo cal search after SP-dec. 16 Per turbed Messa ge P assing f or CSP iterations. Decimation-based metho ds use a maximum of T = 1000 iterations p er iteration of decimation. If any of the metho ds failed to find a solution in the first attempt, T was increased b y a factor of 4 at most 3 times (so in the final attempt: T = 64 , 000 ). T o av oid blo w-up in run-time, for BP-dec and SP-dec, only the maxim um iteration, T , during the first iteration of decimation, was increased (this is similar to the setting of Braunstein et al. (2002) for SP-dec). F or b oth v ariations of SP-dec (see Section 3.1.1), after eac h decimation step, if max i , x i µ ( x i ) − 1 | X i | < . 01 we consider the instance para-magnetic, and run BP-dec (with T = 1000 ,  = . 001 and ρ = 1 %) on the simplified instance. Figure 4(first row) visualizes the success rate of differen t metho ds on 100 instances of 3-SA T (right) and 3-COL (left). Figure 4(second row) rep orts the n umber of v ariables that are fixed b y SP-dec(C) and (S) b efore calling BP-dec as local search. The third ro w shows the av erage amount of time that is used to find a satisfying solution. This do es not include the failed attempts. F or SP-dec v ariations, this time includes the time used b y lo cal searc h. The final ro w of Figure 4 sho ws the n um b er of iterations used by eac h metho d at eac h lev el of difficulty o ver the successful instances. Here the area of each disk is prop ortional to the frequency of satisfied instances with that particular num b er of iterations for eac h control parameter and inference metho d 11 . Here we mak e the follo wing observ ations: • Perturbed BP is muc h more effectiv e than BP-dec, while remaining ten to hun- dreds of times more efficient. • As the con trol parameter grows larger, the chance of requiring more iterations to satisfy the instance increases for all metho ds. • Although computationally very inefficient, BP-dec is able to find solutions for instances with larger control parameters than suggested b y previous results ( e.g. , Mézard and Montanari, 2009). • F or many instances where SP-dec(C) and (S) use few iterations, the v ariables are fixed to a trivial cluster y i = X i , which allows all assignmen ts. This is particularly pronounced for 3-COL, where up to α = 4 . 4 the non-trivial fixes remains zero and therefore the success rate up to this p oint is solely due to BP-dec. • While for 3-SA T, SP-dec(C) and SP-dec(S) hav e a similar p erformance, for 3-COL, SP-dec(C) significan tly outp erforms SP-dec(S). T able 1 rep orts the success-rate as w ell as the av erage of total iterations in the suc c essful attempts of each method. Here the n um b er of iterations for SP-dec(C) and (S) is the sum of iterations used b y the method and the following local search. W e observe that Perturbed BP can solve most of the easier instances using only T = 1000 iterations ( e.g. , see Perturb BP’s result for 3-SA T at α = 4 . 1 , 3-COL at α = 4 . 2 and 9-COL at α = 33 . 4 ). T able 1 also supp orts our sp eculation in Section 3.1.1 that SP-dec(C) is in general preferable to SP-dec(S) , in particular when applied to the coloring problem. The most imp ortan t adv antage of P erturb ed BP ov er SP-dec and P erturb ed SP is that P erturb ed BP can be applied to instances with large factor cardinalit y ( e.g. , 10 -SA T) and large v ariable domains ( e.g. , 9 -COL). F or example for 9 -COL, the cardinalit y of each SP message is 2 9 = 512 , which mak es SP-dec and Perturbed SP impractical. Here BP- dec is not even able to solve a single instance around the dynamical transition (as low as α = 33 . 4 ) while P erturb ed BP satisfies all instances up to α = 34 . 1 . 12 Besides the 11. The n umber of iterations are rounded to the closest p ow er of t wo. 12. Note that for 9 -COL condensation transition happ ens after rigidity transition. So if w e were able to find solutions after rigidit y , it w ould hav e implied that condensation transition marks the onset of difficulty . Ho wev er, this did not o ccur and similar to all other cases, P erturb ed BP failed b efore rigidity transition. 17 Ra v anbakhsh and Greiner exp erimen tal results rep orted here, w e hav e also used p erturbed BP to efficiently solve other CSPs such as K-P ac king, K-set-cov er and clique-cov er within the con text of min-max inference (Rav anbakhsh et al., 2014). T able 1: Comparison of differen t methods on { 3 , 4 } -SA T and { 3 , 4 , 9 } -COL . F or each metho d the success-rate and the av erage num b er of iterations (including lo cal search) on successful attempts are rep orted. The approximate lo cation of phase transitions are from (Mon tanari et al., 2008; Zdeb orov a and Krzak ala, 2007) . BP-dec SP-dec(C) SP-dec(S) Perturbed BP Perturbed SP Problem ctrl param α avg. iters. success rate avg. iters. success rate avg. iters. success rate avg. iters. success rate avg. iters. success rate 3-SA T 3.86 dynamical and condensation transition 4.1 85405 99% 102800 100% 96475 100% 1301 100% 1211 100% 4.15 104147 83% 118852 100% 111754 96% 5643 95% 1121 100% 4.2 93904 28% 118288 65% 113910 64% 19227 53% 3415 87% 4.22 100609 12% 112910 33% 114303 36% 22430 28% 8413 69% 4.23 123318 5% 109659 36% 107783 36% 18438 16% 9173 58% 4.24 165710 1% 126794 23% 118284 19% 29715 7% 10147 41% 4.25 N/A 0% 123703 9% 110584 8% 64001 1% 14501 18% 4.26 37396 1% 83231 6% 106363 5% 32001 3% 22274 11% 4.268 satisfiability transition 4-SA T 9.38 dynamical transition 9.547 condensation transition 9.73 134368 8% 119483 32% 120353 35% 25001 43% 11142 86% 9.75 168633 5% 115506 15% 96391 21% 36668 27% 9783 68% 9.78 N/A 0% 83720 9% 139412 7% 34001 12% 11876 37% 9.88 rigidity transition 9.931 satisfiability transition 3-COL 4 dynamical and condensation transition 4.2 24148 93% 25066 94% 24634 94% 1511 100% 1151 100% 4.4 51590 95% 52684 89% 54587 93% 1691 100% 1421 100% 4.52 61109 20% 68189 63% 54736 1% 7705 98% 2134 98% 4.56 N/A 0% 63980 32% 13317 1% 28047 65% 3607 99% 4.6 N/A 0% 74550 2% N/A 0% 16001 1% 18075 81% 4.63 N/A 0% N/A 0% N/A 0% 48001 3% 29270 26% 4.66 rigidity transition 4.66 N/A 0% N/A 0% N/A 0% N/A 0% 40001 2% 4.687 satisfiability transition 4-COL 8.353 dynamical transition 8.4 64207 92% 72359 88% 71214 93% 1931 100% 1331 100% 8.46 dynamical transition 8.55 77618 13% 60802 13% 62876 9% 3041 100% 5577 100% 8.7 N/A 0% N/A 0% N/A 0% 50287 14% N/A 0% 8.83 rigidity transition 8.901 satisfiability transition 9-COL 33.45 dynamical transition 33.4 N/A 0% N/A N/A N/A N/A 1061 100% N/A N/A 33.9 N/A 0% N/A N/A N/A N/A 3701 100% N/A N/A 34.1 N/A 0% N/A N/A N/A N/A 12243 100% N/A N/A 34.5 N/A 0% N/A N/A N/A N/A 48001 6% N/A N/A 35.0 N/A 0% N/A N/A N/A N/A N/A 0% N/A N/A 39.87 rigidity transition 43.08 condensation transition 43.37 satisfiability transition 3.4 Discussion It is easy to chec k that, for m = 1 , SP up dates pro duce sum-pro duct BP messages as an a v erage case; that is, the SP up dates (Eqs 21 and 22) reduce to that of sum-pro duct BP 18 Per turbed Messa ge P assing f or CSP Figure 5: This sc hematic view demonstrates the clustering during condensation phase. Here assume x and y axes corresp ond to x 1 and x 2 . Considering the whole space of assignments, x 1 and x 2 are highly correlated. The formation of this correlation b etw een distan t v ariables on a PGM breaks BP . No w assume that P erturbed BP messages are fo cused on the largest shaded ellipse. In this case the correlation is significan tly reduced. (Eqs 3 and 4) where ν i → I ( x i ) ∝ X y i 3 x i ν i → I ( y i ) (28) This suggests that the BP equation remains correct wherever SP( 1 ) holds, whic h has lead to the b elief that BP-dec should p erform well up to the condensation transition (Krza- k ala et al., 2007). How ever in reac hing this conclusion, the effect of decimation w as ignored. More recent analyses (Co ja-Oghlan, 2011; Montanari et al., 2007; Ricci-T ersenghi and Se- merjian, 2009) draw a similar conclusion ab out the effect of decimation: A t some point during the decimation pro cess, v ariables form long-range correlations such that fixing one v ariable may imply an assignmen t for a p ortion of v ariables that form a lo op, potentially leading to contradictions. Alternatively the same long-range correlations result in BP’s lac k of con v ergence and error in marginals that ma y lead to unsatisfying assignmen ts. P erturb ed BP a v oids the pitfalls of BP-dec in tw o wa ys: (I) Since man y configurations ha v e non-zero probability until the final iteration, Perturbed BP can av oid contradictions b y adapting to the most recent choices. This is in contrast to decimation in which v ariables are fixed once and are unable to change afterwards. A backtrac king sc heme suggested by P arisi (2003) attempts to fix the same problem with SP-dec. (I I) W e speculate that sim ultaneous bias of all messages tow ards sub-regions o v er whic h the BP equations remain v alid, preven ts the formation of long-range correlations b etw een v ariables that breaks BP in 1sRSB; see Figure 5. In all experiments, we observ ed that P erturbed BP is comp etitiv e with SP-dec, while BP-dec often fails on muc h easier problems. W e saw that the cost of each SP up date gro ws exp onen tially faster than the cost of each BP up date. Meanwhile, our p erturbation scheme adds a negligible cost to that of BP – i.e. , that of sampling from these lo cal marginals and up dating the outgoing messages accordingly . Considering the computational complexity of SP-dec, and also the limited setting under whic h it is applicable, P erturb ed BP is an attractiv e substitute. F urthermore our exp erimental results also suggest that Perturbed SP(S) is a viable option for real-world CSPs with small v ariable domains and constraint cardinalities. 19 Ra v anbakhsh and Greiner Conclusion W e considered the challenge of efficiently pro ducing assignments that satisfy hard com bi- natorial problems, suc h as κ -SA T and q -COL. W e fo cused on wa ys to use message passing metho ds to solve CSPs, and in tro duced a nov el approach, P erturb ed BP , that com bines BP and GS in order to sample from the set of solutions. W e demonstrated that P erturb ed BP is significantly more efficient and successful than BP-dec. W e also demonstrated that P erturb ed BP can b e as p ow erful as a state-of-the-art algorithm (SP-dec), in solving rCSPs while remaining tractable for problems with large v ariable domains and factor cardinali- ties. F urthermore w e provided a metho d to apply the similar p erturbation pro cedure to SP , pro ducing the Perturbed SP pro cess that outp erforms SP-dec in solving difficult rCSPs. A ckno wledgmen t W e would like to thank anon ymous reviewers for their constructive and insigh tful feedbac k. R G receiv ed support from NSERC and Alberta Inno v ate Cen ter for Machine Learning (AICML). Research of SR has b een supp orted by Alb erta Inno v ates T echnology F utures, AICML and QEI I graduate sc holarship. This research has been enabled by the use of computing resources pro vided by W estGrid and Compute/Calcul Canada. References D Ac hioptas and Gregory B Sorkin. Optimal my opic algorithms for random 3-sat. In F oundations of Computer Scienc e, 2000. Pr o c e e dings. 41st Annual Symp osium on , pages 590–600. IEEE, 2000. D Ac hlioptas and A Co ja-Oghlan. Algorithmic barriers from phase transitions. , Marc h 2008. D A c hlioptas, A Naor, and Y Peres. Rigorous location of phase transitions in hard optimization problems. Natur e , 435(7043):759–764, June 2005. ISSN 0028-0836. doi: 10.1038/nature03602. C Andrieu, N de F reitas, A Doucet, and M Jordan. An introduction to MCMC for mac hine learning. Machine L e arning , 2003. A Braunstein and R Zecc hina. Surv ey propagation as lo cal equilibrium equations. CoRR , 2003. A Braunstein, M Mézard, and R Zecc hina. Surv ey propagation: an algorithm for satisfiabilit y . R andom Structur es and Algorithms , 27(2):19, 2002. A Braunstein, R Mulet, A Pagnani, M W eigt, and R Zecchina. Polynomial iterativ e algorithms for coloring and analyzing random graphs. Physic al R eview E , 68(3):036702, 2003. P Cheeseman, B Kanefsky , and W M T aylor. Where the really hard problems are. In Pr o c e e dings of the 12th IJCAI - V olume 1 , IJCAI’91, pages 331–337, San F rancisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1-55860-160-0. S Co cco, O Dub ois, J Mandler, and R Monasson. Rigorous decimation-based construction of ground pure states for spin-glass mo dels on random lattices. Physic al r eview letters , 90(4):047205, 2003. A Co ja-Oghlan. On belief propagation guided decimation for random k-sat. In Pr o c e e dings of the Twenty-Se c ond A nnual ACM-SIAM Symp osium on Discr ete Algorithms , SODA ’11, pages 957–966. SIAM, 2011. 20 Per turbed Messa ge P assing f or CSP V. Deolalik ar. Deolalik ar P vs NP paper, 2010. URL http://michaelnielsen.org/polymath1/ index.php?title=Deolalikar_P_vs_NP_paper . Y F u and P W Anderson. Application of statistical mechanics to NP-complete problems in combi- natorial optimisation. Journal of Physics A: Mathematic al and Gener al , 19(9):1605–1620, June 1986. ISSN 0305-4470, 1361-6447. doi: 10.1088/0305- 4470/19/9/033. M R Garey and D S Johnson. Computers and Intr actability: A Guide to the The ory of NP- Completeness , volume 44 of A Series of Bo oks in the Mathematic al Scienc es . W. H. F reeman, 1979. ISBN 0716710455. A T Ihler and D A Mcallester. Particle b elief propagation. In International Confer enc e on Artificial Intel ligenc e and Statistics , pages 256–263, 2009. S Janson, T Luczak, and V. F. Kolc hin. Random graphs. Bul l. L ondon Math. So c , 33:363–383, 2001. K Kask, R Dec h ter, and V Gogate. Counting-based lo ok-ahead schemes for constrain t satisfaction. In Principles and Pr actic e of Constr aint Pr o gr amming–CP 2004 , pages 317–331. Springer, 2004. H A. Kautz, A Sabharwal, and B Selman. Incomplete algorithms. In Armin Biere, Marijn Heule, Hans v an Maaren, and T oby W alsh, editors, Handb o ok of Satisfiability , volume 185 of F r ontiers in A rtificial Intel ligenc e and Applic ations , pages 185–203. IOS Press, 2009. ISBN 978-1-58603-929-5. S Kirkpatrick and B Selman. Critical b ehavior in the satisfiability of random b o olean expressions. Scienc e , 264(5163):1297–1301, 1994. L Kro c, A Sabharwal, and B Selman. Message-passing and lo cal heuristics as decimation strategies for satisfiability . In Pr o c e e dings of the 2009 ACM symp osium on Applie d Computing , pages 1408– 1414. ACM, 2009. Luk as Kro c, Ashish Sabharwal, and Bart Selman. Surv ey propagation revisited. In In 23r d UAI , pages 217–226, 2007. F Krzak ala, A Montanari, F Ricci-T ersenghi, G Semerjian, and L Zdeb orová. Gibbs states and the set of solutions of random constrain t satisfaction problems. Pr o c e e dings of the National A c ademy of Scienc es , 104(25):10318–10323, 2007. F rank R Kschisc hang, Brendan J F rey , and H-A Lo eliger. F actor graphs and the sum-product algorithm. Information The ory, IEEE T r ansactions on , 47(2):498–519, 2001. Ch Lecoutre. A collection of csp b enc hmark instances., Octob er 2013. URL http://www.cril. univ- artois.fr/~lecoutre/benchmarks.html . E Manev a, E Mossel, and M W ainwrigh t. A new lo ok at survey propagation and its generalizations. Journal of the A CM (JACM) , 54(4):17, 2007. M. Mézard and A. Mon tanari. Information, Physics, and Computation . Oxford, 2009. M Mézard and G Parisi. The Bethe lattice spin glass revisited. The Eur op e an Physic al Journal B-Condense d Matter and Complex Systems , 20(2):217–233, 2001. M Mézard and G P arisi. The cavit y metho d at zero temp erature. J. Stat. Phys 111 1 , 2003. M Mézard, G Parisi, and M A Virasoro. Spin Glass The ory and Beyond . Singap ore: W orld Scientific, 1987. 21 Ra v anbakhsh and Greiner M Mézard, G P arisi, and R Zecc hina. Analytic and algorithmic solution of random satisfiability problems. Scienc e , 297(5582):812–815, August 2002. ISSN 0036-8075, 1095-9203. doi: 10.1126/ science.1073287. D Mitchell, B Selman, and H Levesque. Hard and easy distributions of sat problems. In Asso ciation for the A dvanc ement of A rtificial Intel ligenc e (AAAI) , volume 92, pages 459–465, 1992. A Mon tanari, F Ricci-tersenghi, and G Semerjian. Solving constrain t satisfaction problems through b elief propagation-guided decimation. In in Pr o c. of the Al lerton Conf. on Commun., Contr ol, and Computing , 2007. A Montanari, F Ricci-T ersenghi, and G Semerjian. Clusters of solutions and replica symmetry breaking in random k-satisfiabilit y . , F ebruary 2008. doi: 10.1088/1742- 5468/ 2008/04/P04004. J. Stat. Mech. P04004 (2008). J M Mo oij. libDAI: A free and open source C++ library for discrete appro ximate inference in graphical mo dels. Journal of Machine L e arning R ese ar ch , 11:2169–2173, August 2010. R. Mulet, A. P agnani, M. W eigt, and R. Zecchina. Coloring random graphs. Physic al r eview letters , 89(26):268701, 2002. N No orshams and M J. W ainwrigh t. Belief propagation for con tinuous state spaces: Sto chastic message-passing with quantitativ e guarantees. Journal of Machine L e arning R ese ar ch , 14:2799– 2835, 2013. G Parisi. A backtrac king survey propagation algorithm for k-satisfiability . Arxiv pr eprint c ond- mat0308510 , page 9, 2003. J Pearl. Pr ob abilistic R e asoning in Intel ligent Systems: Networks of Plausible Infer enc e , volume 88 of R epr esentation and R e asoning . Morgan Kaufmann, 1988. ISBN 1558604790. S Rav anbakhsh, C Sriniv asa, B F rey , and R Greiner. Min-max problems on factor-graphs. Pr o c e e dings of The 31st International Confer enc e on Machine L e arning, pp. 1035âĂŞ1043 , 2014. F Ricci-T ersenghi. Being glassy without b eing hard to solve. Scienc e , 330(6011):1639–1640, 2010. F Ricci-T ersenghi and G Semerjian. On the cavit y metho d for decimated random constraint satisfaction problems and the analysis of belief propagation guided decimation algorithms. arXiv:0904.3395 , April 2009. J Stat. Mech. P09001 (2009). Ch P . Rob ert and G Casella. Monte Carlo Statistic al Metho ds (Springer T exts in Statistics) . Springer-V erlag New Y ork, Inc., Secaucus, NJ, USA, 2005. ISBN 0387212396. O Roussel and Ch Lecoutre. Xml represen tation of constrain t netw orks: F ormat xcsp 2.1. arXiv pr eprint arXiv:0902.2362 , 2009. B Selman, H A Kautz, and B Cohen. Noise strategies for improving lo cal searc h. In Asso ciation for the A dvanc ement of Artificial Intel ligenc e (AAAI) , v olume 94, pages 337–343, 1994. L G. V aliant. The complexity of computing the p ermanent. The or etic al c omputer scienc e , 8(2): 189–201, 1979. L Zdeb orov a and F Krzak ala. Phase transitions in the coloring of random graphs. Physic al R eview E , 76(3):031131, 2007. 22 Per turbed Messa ge P assing f or CSP A. Detailed Results for Benc hmark CSP In T able 2 we rep ort the a verage num ber of iterations and av erage time for the attempt that successfully found a satisfying assignment ( i.e. , the failed instances are not included in the a v erage). W e also rep ort the num b er of satisfied instances for each metho d as w ell as the n um b er of satisfiable instance in that series of problems (if known). F urther information ab out each data-set maybe obtained from Lecoutre (2013). 23 Ra v anbakhsh and Greiner T able 2: Comparison of Perturbed BP and BP-guided decimation on b enchmark CSPs. BP-dec Perturbed BP problem series instances # satisfiable # satisfied avg. time (s) avg. iters # satisfied avg. time (s) avg. iters Geometric - 100 92 77 208.63 30383 81 .70 74 Dimacs aim-50 24 16 9 11.41 25344 14 .07 181 aim-100 24 16 8 18.2 16755 11 .15 213 aim-200 24 N/A 7 401.90 160884 6 .17 46 ssa 8 N/A 4 .60 373.25 4 .50 86 jhnSat 16 16 16 5839.86 141852 13 9.82 117 v arDimacs 9 N/A 4 2.95 715 4 .12 18 QCP QCP-10 15 10 10 43.87 30054 10 .22 51 QCP-15 15 10 3 5659.70 600741 4 9.59 530 QCP-25 15 10 0 0 0 0 0 0 Graph-Coloring ColoringExt 17 N/A 4 .05 103 5 .04 25 school 8 N/A 0 N/A N/A 5 62.86 153 myciel 16 N/A 5 .21 59 5 .05 11 hos 13 N/A 5 27.34 606 5 10.04 37 mug 8 N/A 4 .068 313 4 .004 11 register-fpsol 25 N/A 0 N/A N/A 0 N/A N/A register-inithx 25 N/A 0 N/A N/A 0 N/A N/A register-zeroin 14 N/A 3 5906.16 26544 0 N/A N/A register-mulsol 49 N/A 5 59.27 418 0 N/A N/A sgb-queen 50 N/A 7 35.66 916 11 7.56 81 sgb-games 4 N/A 1 .91 434 1 .07 21 sgb-miles 34 N/A 4 20.86 371 2 4.20 181 sgb-bo ok 26 N/A 5 1.72 444 5 .18 39 leighton-5 8 N/A 0 N/A N/A 0 N/A N/A leighton-15 28 N/A 0 N/A N/A 1 106.46 641 leighton-25 29 N/A 2 304.49 1516 2 94.11 241 All Interv al Series series 12 12 2 4.78 11319 7 1.85 520 Job Shop e0ddr1 10 10 9 707.74 9195 5 37 257 e0ddr2 10 10 5 3640.40 26544 7 74.49 366 ewddr2 10 10 10 10871.96 48053 9 21.24 72 Sch urr’s Lemma - 10 N/A 1 39.89 120152 2 .97 100 Ramsey Ramsey 3 8 N/A 1 .01 61 4 .75 283 Ramsey 4 8 N/A 2 12941.51 561300 7 7.39 81 Chessboard Coloration - 14 N/A 5 35.51 3111 5 .66 27 Hanoi - 3 3 3 .48 12 3 .52 14 Golomb Ruler Arity 3 8 N/A 2 1.39 103 2 19.78 660 Queens queens 8 8 7 3.30 159 8 2.43 57 Multi-Knapsack mknap 2 2 2 2.44 6 2 4.41 10 Driver - 7 7 5 10.14 1438 5 4.74 274 Composed 25-10-20 10 10 8 1.62 695 5 .17 38 Langford lagford-ext 4 2 0 N/A N/A 1 .002 10 lagford 2 22 N/A 4 .67 127 10 11.64 10 lagford 3 20 N/A 0 N/A N/A N/A N/A N/A 24