Core Persistence in Peer-to-Peer Systems: Relating Size to Lifetime

Cor e Pe rsistence in P eer -to-Peer Systems: Relating Size to Lifetime V incent Gramoli, Anne-Marie K erm arrec Achour Mostefaoui, Michel Raynal, and Bruno Sericola IRISA (Univ ersit ´ e de Rennes 1 and INRIA) Campus de Beaulieu, 3504 2 Rennes, Fr ance. { vgramoli,akerma rr,achour,rayn al,bsericol } @irisa.fr Abstract. Distributed systems are no w both very large and highly dynamic. Peer to peer ov erlay networks hav e been prov ed efficient to cope with this new deal that traditional a pproaches can no longer accommo date. While the challenge of organ izing peers in an ov erlay network has generated a lot of interest leading to a large n umber of solutions, maintaining critical da ta in such a network remains an open issue. In this paper , we are interested in defining the portion of nodes and frequenc y one has to probe, gi ven the churn observ ed in the system, in order to achie ve a gi ven probability of maintaining the persistence of some critical data. More specifically , we pro vide a clear result relating the size an d the frequenc y of the p r obing set a long with its proof as well as an analysis of the way of le veraging such an information in a large scale dyn amic distributed system. K eywords : Churn, Core, Dy namic system, Peer to peer system, Persistence, Prob- abilistic guarantee, Quality of service, Surviv ability . 1 Intr oduction Context of the pa per . Persistence o f critical data in distributed applications is a crucial problem . Althoug h static systems h av e experienced many so lutions, mo stly r elying o n defining the right degree of replication, this remains an open issue in the context of dynamic systems. Recently , peer to pe er (P2P) systems became po pular as they h av e been p roved efficient to cope with the scale shift observed in distrib uted systems. A P2P system is a dynamic system th at a llow peers (nod es) to join or leave th e sy stem. In the mean time, a natural tendency to trade strong deterministic guarantees for probabilistic ones aimed at coping with both scale and dynamism. Y et, quantifyin g bounds of guarantee that can be achieved probabilistically is very important for the deploymen t of applications. More specifically , a typ ical issue is to ensure th at despite dynam ism some critical data is not lost. The set of nodes owning a copy of the critical data is called a cor e (distinct cores can possibly co-exist, each associated with a particular data). Provided that core nodes remain lon g en ough in the system, a “data/state transfer” protoco l can tran smit the c ritical data from nodes to nod es. Th is en sures that a new core of nodes in the system will keep tr ack of the data. Hen ce, such pr otocols provide data persistence despite the uncertainty of the system state inv olved by the dyn amic ev olutio n of its members. There is howe ver an inher ent tradeoff in th e use of such pr otocols. If the policy that is used is too con servati ve, the data tr ansfer pro tocol migh t be executed too often, thereby consuming resources and increasing the whole system overhead. Con versely , if the proto col is executed too rarely , all n odes owning a copy of the data may leav e (or crash) bef ore a new pro tocol execution, an d the data would be lost. Th is fun damental tradeoff is the main prob lem addressed in this paper . Content of the pap er . Considering the previous context, we are interested in providing some p robabilistic guar antees of ma intaining a cor e in the system. Mo re precisely , g iv en the churn observed in the system, we aim at maintaining the per sistence of som e c ritical data. T o this end, we are interested in defining the portion of n odes that must be p robed, as well as the freq uency to which this pr obe must oc cur to achieve th is result with a giv en probab ility . This boils down to relating the size an d the frequ ency of the pr obing set accordin g to a target probab ility of success and the churn observed in the system. The in vestigation of the previous tradeoff relies on critical parameters. One of them is n aturally the size of the c ore. T w o o ther para meters are the per centage of n odes that enter/leave the system per time unit, and the duratio n during wh ich we observe the system. W e first assume that, per time unit, the number of entering nodes is the same as the number of leaving nodes. In other w ords, the number of nodes remains constant. Let S be the system at some time τ . It is composed of n n odes includ ing a subset of q nodes defining a core Q for a given critica l data. Let S ′ be the system a t time τ + δ . Becau se of the system e volution , some nodes owning a copy of the critical d ata at time τ m ight have left the system at time τ + δ (tho se nodes are in S and no t in S ′ ). So, an important que stion is the following: “Giv e n a set Q ′ of q ′ nodes of S ′ , what is the p robab ility th at Q and Q ′ do inter sect?” W e der i ve an explicit expression of this probab ility as a fu nction of the parameters character izing the d ynamic system. This allows us to compute some of them wh en other ones are fixed. This p rovides d istributed applications with the opportun ity to set a tradeoff between a probab ilistic guarantee of achieving a core and the overhead inv olved compu ted either as the number of nodes probed or the frequency at which the probing set needs to be refreshed. Related work. As m entioned above, P2P systems have receiv e d a g reat deal of atten - tion bo th in aca demia and in dustry for the past five years. Mo re specifically , a lot of approa ches hav e been proposed to c reate whethe r they are structured, such as Chord [18], CAN [15] or Pastry [16], or unstructured [5,6,9]. Maintenan ce of such overlay networks in the presence of high ch urns has also bee n studied as o ne of th e major goal of P2P overlay network s [10]. The paramete rs impactin g on con nectivity and routing capabilities in P2P overlay networks are now well understood . In structu red P2P networks, ro uting tables con tain critical info rmation and ref resh- ment must occur with some f requen cy depe nding on the ch urn observed in the net- work [2] to achieve routin g cap abilities. For instance in Pastry , the size o f the leaf set (set of nodes whose id entities are numerica lly the closest to cu rrent n ode iden tity) and its mainten ance pro tocol can be tuned to achiev e the routing within reasonable delay stretch an d low overhead . Finally , there has been a pproach es e valuating the number o f locations to wh ich a data has to be replicated in the system in order to be successfu lly searched by flooding-based o r random walk-based algorithms [4]. These approaches d o not con sider specifically chur n in their analy sis. In this paper churn is a primary concern. The result o f this work can be ap plied to any P2P network, regardless of its struc ture, in order to maintain critical data by refreshmen t at sufficiently many locations. The use of a base co re to extend pro tocols d esigned for static systems to d ynamic systems has been inves tigated in [14]. Persistent cor es shar e some features with quo- rums (i.e., mutually intersecting sets). Quorums originated a long time ago with major - ity voting systems [7,19] introduced t o ensure data consistency . Mo re recently , quorum reconfigu ration [1 1,3] hav e been prop osed to face system dynam ism wh ile guaran tee- ing atomic consistency: this application outlines the strength of such dynamic quorums. Quorum -based p rotocols for sear ching objects in P2 P systems are p roposed in [ 13]. Probabilistic quorum systems hav e been introduced in [12]. They use randomizatio n to relax the strict intersection proper ty to a probab ilistic one. They ha ve been extended to dynamic systems in [1]. Roadma p. The paper is o rganized as fo llows . Section 2 defin es th e system model. Section 3 d escribes our dynamic system analysis an d our pr obabilistic r esults. Section 4 interprets the previous formulas and sh ows how to use them to control th e uncertainty of the key parameters of P2P applications. Finally , Section 5 concludes the paper . 2 System model The system mode l, sketched in the in troduction is simple. The system co nsists of n nodes. It is d ynamic in the following sense. For the sake of simplicity , let n b e the size of th e system. Every time unit, cn no des leave the system an d cn nodes enter the system, where c is the percen tage of nodes that enter/leave the system per time unit; this can be seen as new no des “replacing” leaving n odes. Although monitorin g the leave a nd join rates of a large-scale dynamic s y stem remains an open issue, it is reasonable to assume join and lea ve are tightly correlated in P2P systems. A more realistic model w o uld take in account variation of the system size depe nding for instance, on night-time an d d ay- time as observed in [17]. A node lea ves the system either voluntarily or because it crashes. A node that le a ves the system do es not en ter it later . (Practically , this means that, to r e-enter the system, a nod e th at has left mu st be considered as a n ew no de; all its previous kn owledge of the sy stem state is lost.) For instance, initially (at tim e τ ), assume there are n nodes (identified from 1 to n ; let us take n = 5 to simplify). L et c = 0 . 2 , which means that, ev e ry time un it, nc = 1 no de ch anges ( a node d isappears and a new n ode replaces it). That is, at time τ + 1 , one node leaves the system an d ano ther one joins. From now on, observe that next leaving nodes are eith er node s that were initially in th e system o r nodes that joined after time τ . 3 Relating the key parameters of the dynamic system This section answer s th e question posed in th e introdu ction, nam ely , given a set Q ( τ ) of n odes at time τ (th e cor e), and a set Q ( τ ′ ) o f no des at time τ ′ = τ + δ , what is the probab ility of the e vent “ Q ( τ ) ∩ Q ( τ ′ ) 6 = ∅ ” . In the remaining o f this paper , we assume that both Q ( τ ) and Q ( τ ′ ) con tain q nod es, since an interesting goal is to minimize both the n umber of n odes where the data is replicated and the nu mber of n odes one h as to probe to find the data. Let an initia l n ode be a node that b elongs to th e system at tim e τ . Moreover , without loss of generality , let τ = 0 (hence , τ ′ = δ ). Lemma 1. Let C be the ratio of initial n odes that are r eplaced after δ time u nits. W e have C = 1 − (1 − c ) δ . Proof W e claim that the number of initial nodes that are still in the system after δ time units is n (1 − c ) δ . The proof is by induction on the time instants. Let us remind that c is the percen tage of nodes that are replaced in one time unit. For the Base case, at time 1 , n − nc = n (1 − c ) nodes hav e not been replaced. For the induction case, let us assume that at time δ − 1 , the number of initial nod es th at ha ve not b een replaced is n (1 − c ) δ − 1 . Let us consider the time in stant δ . The nu mber of initial nod es that are not replaced after δ time units is n (1 − c ) δ − 1 − n (1 − c ) δ − 1 c , i.e., n (1 − c ) δ , which p roves the claim. It follows from the previous claim th at the number o f initial nod es that are replace d during δ time un its is n − n (1 − c ) δ . Hence, C = ( n − n (1 − c ) δ ) /n = 1 − (1 − c ) δ . ✷ Lemma 1 Giv en a core of q nodes at time τ (each having a copy o f the critical data), the following the orem gives the pr obability that, at time τ ′ = τ + δ , an ar bitrary nod e cannot obtain the data when it queries q nodes arbitrarily chosen. For th is purpo se, using result of Lemma 1 we take th e number of elements that hav e left the system d uring the period δ as α = ⌈ C n ⌉ = ⌈ (1 − (1 − c ) δ ) n ⌉ . Th is numb er allows us to e valuate the aforementioned probability . Theorem 1. Let x 1 , ..., x q be any no de in the system at time τ ′ = τ + δ . The pr obability that none of these nodes belong to the initial cor e is P b k = a  n + k − q q  q k  n − q α − k   n q  n α  , wher e α = ⌈ (1 − (1 − c ) δ ) n ⌉ , a = max(0 , α − n + q ) , an d b = min( α, q ) . Proof The problem we ha ve to solve can be represented in the following way: The system is an urn containing n b alls (nodes), such th at, initially , q b alls are green (they represent th e initial co re Q ( τ ) and are represented by the set Q in Figure 1) , while the n − q remaining balls are black. W e rando mly draw α = ⌈ C n ⌉ balls f rom the urn (accordin g to a uniform distribu- tion), an d paint th em re d. These α balls r epresent the initial nod es that ar e replaced b y new nodes after δ units o f time (each o f these balls was initially gre en or black). Afte r it has bee n colored red, each of these ba lls is put back in the u rn (so, the urn conta ins again n b alls). W e then ob tain the system as descr ibed in the right part of Fig ure 1 (which repre- sents th e system state at time τ ′ = τ + δ ). The set A is the set of balls that hav e bee n painted red. Q ′ is the co re set Q after some of its balls ha ve been painted red (these balls represent th e no des of the cor e that h a ve left the system). This mean s the set Q ′ \ A , that we denote by E , contains all the green balls and only them. W e deno te by β the numb er of balls in the set Q ′ ∩ A . It is well-known that β has a hypergeo metric distrib ution, i.e., for a ≤ k ≤ b whe re a = max(0 , α − n + q ) an d b = min( α, q ) , we h av e Pr[ β = k ] =  q k  n − q α − k   n α  . (1) W e finally draw r andomly and successi vely q balls x 1 , ..., x q from the urn ( system at time τ ′ ) without replac ing them. The problem consists in co mputing the pro bability of the ev e nt { none of the selected balls x 1 , ..., x q are g reen } , which c an be written as Pr[ x 1 / ∈ E , ..., x q / ∈ E ] . 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 000000000 000000000 000000000 000000000 000000000 000000000 000000000 000000000 000000000 000000000 111111111 111111111 111111111 111111111 111111111 111111111 111111111 111111111 111111111 111111111 A Q The system at time τ The system at time τ ′ Q ′ Fig. 1. The system at times τ and τ ′ = τ + δ As { x ∈ E } ⇔ { x ∈ Q ′ } ∩ { x / ∈ Q ′ ∩ A} , we have (taking the co ntrapositive) { x / ∈ E } ⇔ { x / ∈ Q ′ } ∪ { x ∈ Q ′ ∩ A} , from which we can conclude Pr[ x / ∈ E ] = Pr[ { x / ∈ Q ′ } ∪ { x ∈ Q ′ ∩ A} ] . As the e vents { x / ∈ Q ′ } a nd { x ∈ Q ′ ∩ A} are disjoints, we obtain Pr[ x / ∈ E ] = Pr[ x / ∈ Q ′ ] + Pr[ x ∈ Q ′ ∩ A ] . T he system contains n balls. T he n umber o f b alls in Q ′ , A and Q ′ ∩ A is equ al to q , α and β , respectively . Since there is no replacemen t, we get, Pr[ x 1 / ∈ E , ..., x q / ∈ E ‹ β = k ] = P b k = a Q q i =1 “ 1 − q − k n − i +1 ” = P b k = a n − q + k q ! n q ! . (2) T o u nconditio n the aforementio ned result (2), we simply multiply it by (1), leading to Pr[ x 1 / ∈ E , ..., x q / ∈ E ] = P b k = a  n + k − q q  q k  n − q α − k   n q  n α  . ✷ T heor em 1 4 Fr om formu las to parameter tuning In the previous section, we h av e provided a set o f formu las th at can b e leveraged an d exploited by distributed applications in many ways. T ypically , in a P2P system, the churn rate is not negotiated but observed 1 . Nevertheless, applications deployed on P2P overlays may need to ch oose the probab ilistic guar antees that a nod e of the initial core is probed . Gi ven su ch a p robability , the ap plication may fix either the size of the probin g set of n odes or th e freque ncy at wh ich the cor e needs to b e re-established fro m the current set of nodes (with the help of an approp riate data transfer protocol). This section exploits th e previous formu la to relate these various elem ents. More precisely , w e p rovide the various relation s existing between the three factors th at can be tu ned b y an ap plication de signer: the size of th e pro bing set q , the frequ ency of the probin g δ , and the pr obability o f ac hieving a co re characterized b y p = 1 − ǫ . ( For the sake of clarity all along this section, a ratio C , or c , is some times expressed as a percentag e. Floating poin t numbe rs on the y -axis are represented in their mantissa and exponent numbers.) Relation linking c and δ . The fir st parameter th at we fo rmalized is C , that can be interpreted as the rate of dynamism in the system. C depend s both on t he churn rate ( c ) observed in th e system a nd th e prob ing frequen cy ( 1 /δ ). More specifically , we foresee here a scenario in which an application designer would consider tolerating a churn C in order to define the size of a core a nd thus ensure the persistence of so me critical data. For example, an ap plication may ne ed to to lerate a churn rate o f 10% in the system, meaning that the persistence o f some critical data sho uld be ensured as lo ng a s up to 10% of the nod es in the system change over tim e. Th erefore, depend ing o n the churn observed and monitored in the system, we are able to define the longest period δ befo re which the c ore shou ld be re-in stantiated on a set o f the curr ent nodes. On e of the main interest of linking c and δ is that if c varies over time, δ can be adapted accord ingly without compr omising the initial requirem ents of the application. More fo rmally , Lemma 1 provid es an exp licit value of C ( the ra tio of initial n odes that are replaced) a s a function of c ( the replacement ratio per time unit) and δ (the number of time units). Figure 2 rep resents this fu nction for se veral v alues of C . More explicitly , it depicts on a logarithm ic scale the curve c = 1 − δ √ 1 − C (or equiv alently , the curve δ = log(1 − C ) log(1 − c ) ). As an example, the curve associated with C = 10% indicates that 10% o f th e initial n odes ha ve been replaced after δ = 105 time units (poin t A, Figure 2), wh en th e replacement r atio is c = 10 − 3 per time u nit. Similar ly , the same replacemen t ratio per time unit entails th e replacement of 30% of the initial nodes when the duration we consider is δ = 35 6 time units (point B, Figure 2). The s y stem designer can benefit from these values to better appreciate the way the system e volves according to the assumed replacemen t r atio per time unit. T o summarize, this result ca n be used as follows. In a system, aiming at tolerating a chu rn of X % of th e nodes, our goal is to provide an ap plication with the corresponding value of δ , knowing the churn c ob served in the system. This giv es the opportun ity to adjust δ if c ch anges over time. 1 Monitoring the churn rate of a system, although very interesting, is out of the scope of this paper . .1e–3 .1e–2 .1e–1 .1 1. 200 400 600 800 1000 c : replaceme nt ratio by time unit δ : num ber of time units C = 10% C = 30% C = 50% C = 70% C = 90% . A . B 0 0 0 0 105 356 Fig. 2. Evolution of the pair ( c, δ ) fo r gi ven v alues of C Relation linking the cor e size q and ǫ Now , gi ven a v a lue C set by an application dev e l- oper, there are still two p arameters that may in fluence either th e overhead of maintaining a core in the system, or the probabilistic guarantee of ha ving such a core. The overhead may be measured in a straigh tforward manner in this co ntext as the n umber of no des that need to be probed, namely q . Intuiti vely , for a gi ven C , as q incr eases, the probabil- ity of pr obing a node o f th e initial core increases. In this sectio n, we defin e how much these parameters are related. Let us consider the v alue ǫ determin ed by Theorem 1. That v alue can be interpreted the following way: p = 1 − ǫ is the proba bility th at, at time τ ′ = τ + δ , on e of the q queries issued (ran domly) by a node hits a n ode of the co re. An important question is then the following: How are ǫ and q related? Or eq uiv alently , how in creasing the si ze of q e nable to decrease ǫ ? This relation is depicted in Figure 3(a) where se veral curves are represented for n = 1 0 , 0 00 nodes. Each curve cor responds to a percentage of the initial nodes that ha ve been replaced. (As an exam ple, th e c urve 30 % corr esponds to the case wher e C = 3 0% of the initial nodes have left the system; the way C , δ and c are related has been seen previously .) Let us consider ǫ = 10 − 3 . Th e curves show that q = 274 is a sufficient core size for not byp assing that value of ǫ when u p to 10% of th e nod es are r eplaced (p oint A, Figure 3(a)) . Differently , q = 274 is not sufficient wh en up to 50% of the nodes are replaced; in that case, the size q = 3 69 is requ ired (point B, Figure 3(a)). The curves o f both Figu re 2 an d Figu re 3(a) pr ovide the system d esigner with realis- tic hints to set t he v alue of δ (deadline be fore which a data transfer pro tocol establishing a new core has to be executed). Figure 3(b) is a zoom of Figure 3(a) focusing on th e small values of ǫ . It shows that, whe n 10 − 3 ≤ ǫ ≤ 10 − 2 , the probability p = 1 − ǫ in- creases very rapidly towards 1, though the size o f the core in creases only very slightly . As an example, let u s consider the cur ve associated with C = 10% in Figur e 3(b). It shows that a core of q = 22 4 nodes ensures an intersection probability = 1 − ǫ = 0 . 9 9 , and a core of q = 274 no des ensures an intersection probability = 1 − ǫ = 0 . 999 . 1e-10 1e-08 1e-06 0.0001 0.01 1 0 200 400 600 800 1000 ǫ : the prob ability 1 − p q : the core size C = 10% C = 30% C = 50% C = 70% C = 90% . A . B 274 369 (a) 0.002 0.004 0.006 0.008 0.01 200 400 600 800 1000 ǫ : the prob ability 1 − p q : the core size C = 10% C = 30% C = 50% C = 70% C = 90% 224 274 (b) Fig. 3. Non-in tersection probability over the core size Interestingly , this phen omenon is similar to the birthday paradox 2 [8] that can be rough ly summar ized a s follows. How many perso ns must be p resent in a room for two of them to have the same birthday with probability p = 1 − ǫ ? Actually , for that probability to be greater than 1 / 2 , it is sufficient that the nu mber of persons in the room be equal (only) to 23 ! Whe n, there are 50 person s in the room, the probability becomes 97 % , and increases to 99 . 9996% for 100 persons. In our case, we observe a similar phenomen on: the prob ability p = 1 − ǫ increases very rapidly despite th e fact that the freq uency o f the core size q increases slightly . In our case, this means that th e system d esigner can choose to slightly increase th e size o f the probing set q (and therefor e only slightly increase th e associated overhead) while significantly increasing the prob ability to access a node of the core. Relation linking q and δ So far , we have co nsidered th at an a pplication m ay need to fix C an d then d efine the size of th e probing set to ach iev e a given probab ility p of success. Th ere is anoth er remaining trade- off that an application de signer migh t want to d ecide upon: trad ing the size of the probin g set with the probing fr equency while fixing the probability p = 1 − ǫ of intersecting the initial core. This is precisely defined by relating q to δ for a fixed ǫ . In the f ollowing we investigate the way the size and lifetime of th e core ar e related when the requ ired in tersection probability is 99% or 99 . 9 % . W e chose these values to better illustrate our purpose, as we belie ve they reflec t wh at could be e x pected b y an ap- plication designer . For both pro babilities we present two different figures summarizing the required values of q . Figure 4 focuses on the core size that is required in a static system and in a dynamic system (ac cording to various values o f the ratio C ). The static system implies tha t no nodes leave or join the system while the dynam ic system co ntains nodes that join and leav e the system d ependin g on se veral ch urn values. F or the sake of c larity we om it 2 The paradox is with respect to intuition, not with respect to logics. Intersection Churn Core size probability C = 1 − (1 − c ) δ n = 10 3 n = 10 4 n = 10 5 static 66 213 677 * 10% 70 224 714 99% 30% 79 255 809 60% 105 337 1071 80% 143 478 1516 static 80 * 260 828 * 10% 85 274 873 * 99 . 9% 30% 96 311 990 * 60% 128 413 * 1311 80% 182 584 1855 Fig. 4. The core size dep ending on the s ystem sizes and the churn rate. values of δ and simp ly present C taking several v alues from 10% to 80% . The analysis of the results depicted in the figure leads to two interesting observations. First, when δ is big enough for 10% of the system nodes to be replaced, then the core size req uired is amazing ly close to the static c ase ( 873 versus 828 when n = 10 5 and the probability is 0 . 999 ). Moreover , q has to be equal to 990 only when C increases up to 30 % . Second , even whe n δ is sufficiently large to let 80% of th e system nodes be replaced, the minimal number of node s to pr obe remains lo w with respect to the system size. For instance, if δ is suffi ciently large to let 6 , 000 n odes b e replaced in a system with 10 , 00 0 nodes, then only 41 3 nodes must be rand omly probed to obtain an intersection with prob ability p = 0 . 999 . T o conclude, these results clear ly show that a critical data in a highly dyn amic sys- tem can p ersist in a s calable w ay : even tho ugh th e d elay between co re re-establishments is reasonably large while the size of the core remains relati vely lo w . 5 Conclusion Maintenance of critical data in large-scale dynamic systems where nodes may join and leav e dynam ically is a critical issue. In th is paper , we define the no tion of persistent core of no des that can maintain such cr itical d ata with a high probab ility regardless of the structure of th e underlying P2P network. More specifically , we relate the parameters that can be tuned to achiev e a high probab ility of defining a core, namely the size of the core, the frequency at which it has to be re-established, a nd the c hurn rate o f the system. Our results provide app lication designers with a set of g uidelines to tune the system parameters depe nding o n the expected gu arantees a nd the ch urn rate variation. An in- teresting outcome of this paper is to sh ow that slightly increasing the size of th e cor e result in a significant probab ility increase of the guarantee . This work opens up a number of very inte resting research d irections. An interesting question is re lated to the design and ev alua tion of efficient probin g protocols, d efining such a core in the system ap plicable to a large spectrum of peer to peer overlay net- works. Mo nitoring the system in ord er to estimate the chu rn rate is another interesting issue. Refer ences 1. I. Abraham and D. Malkhi. Probabilistic quo rum systems for dynamic systems. Distributed Systems , 18(2):113–1 24, 2005. 2. M. Castro, M. Costa, and A. Ro wstron. Perf ormance and depen dability o f structured peer-to- peer o verlays. In Pr oc. In t’l IEEE Con fer ence on Dependa ble Systems and Networks (DSN) , pages 9–18, Florence (Italy), 2004. 3. G. Chockler , S. Gilbert, V . Gramoli, P . M. Musial, and A. A. Shv artsman. Reconfigurable distributed storage for dynamic networks. In Pr oceedings of 9th International Confer ence on Principles of Distributed Systems (OPODIS) , Dec. 2 005. 4. E. Cohen and S. Shenk er . Replication str ategies in unstructured peer-to-peer networks. In Pr oc. of ACM SIGCOMM , pag es 177–190. ACM Press, 2 002. 5. P . T . E ugster , R . Guerraoui, S . B. Handurukande, A.- M. Kermarrec, and P . Kouz netsov . Lightweight probabilistic broadcast. ACM Tr ans. on Comp. Systems , 21(4):341–374, 2003. 6. A. J. Ganesh, A.-M. Kermarrec, and L. Massouli ´ e. Peer-to-peer membership manageme nt for gossip-based protocols. IEEE T rans. on Computers , 52 (2):139–149 , February 2003. 7. D. K. Gifford. W eighted voting for replicated data. In Proc . 7th Int’l ACM Symposium on Operating Systems Principles (SOSP) , pag es 150–162. ACM Press, 1 979. 8. R. Isaac. The pleasur e of pr obabilities . Springer V e rlag, Reading in Mathematics, 1995. 9. M. Jelasity , R. Guerraoui, A.-M. Kermarrec, and M. va n S teen. The peer sampling service: Experimental ev aluation of unstructured gossip-based implementations. In H.-A. Jacobsen, editor , Middlewar e 2004 , v olume 3231 of LNCS , pages 79–98. Springer-V erlag, 2004. 10. D. Liben-No well , H. Balakrishnan, and D. Karger . Analysis of t he e volution of peer-to-pee r systems. I n Pr oceedings of the twenty-first annual symposium on Principles of distri buted computing (PODC) , pages 233–2 42, New Y ork, NY , USA, 2002. A CM Press. 11. N. L ynch and A. Shv artsman. RAMBO: A reconfigurable atomic memory service for dy- namic networks. In Pr oc. of 16th International Sy mposium on Distributed Computing , pages 173–19 0, 2002. 12. D. Malkhi, M. Reit er , A. W ool, and R. Wright. Probabilistic quorum systems. Information and Computation , 170(2):184 –206, 2001. 13. K. Miura, T . T ag aw a, and H. Kakugawa. A quorum-based protocol for searching objects in peer-to-peer networks. IEEE T rans. on P ar allel and Distributed Systems , 17(1):25–37, January 2006. 14. A. Mostefaoui, M. Raynal, C. T ravers, S. Peterson, A. E. Abbadi, and D. Agrawa l. From static distributed systems to dynamic systems. In Pr oc. 24th IEEE Symposium on Reliable Distributed Systems (SRDS) , p ages 109–119, 2005. 15. S. Ratnasamy , P . Francis, M. Hand ley , R. Karp, and S. S chenk er . A scalable content- addressable network. In Proc . of A CM SIGCOMM , pages 161–172, S an Diego, CA, 2001. A CM Press. 16. A. Ro wst ron and P . Druschel. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In R. Guerrao ui, editor , Middlewar e 2001 , volume 2218 of LNCS , pages 329–350 . Springer-V erlag, 2001 . 17. S. Saroiu, P . Gummadi, and S . Gribble. A measurement study of peer-to-peer fil e sharing systems. In In Pr oceedings of Multimedia Computing and Networking , 2002. 18. I. Stoica, R. Morris, D. Karger , M. F . Kaashoe k, and H. Balakrishnan. Chord: A scalable peer-to-pee r lookup service for internet applications. In Proc. of ACM SIGCOMM , pages 149–16 0, San Diego (CA), 2001 . A CM P ress. 19. R. H. Thomas. A majority consensus approach to concurren cy control for multiple copy databases. ACM T r ans. on Database Systems , 4(2):180– 209, 1979. 1e–10 1e–09 1e–08 1e–07 1e–06 1e–05 .1e–3 .1e–2 .1e–1 .1 1. 200 400 600 800 1000 0.002 0.004 0.006 0.008 0.01 0 200 400 600 800 1000