Self-Stabilizing Pulse Synchronization Inspired by Biological Pacemaker Networks

Self-stabi lizing Pulse Synchronization Inspired by Biological Pacemaker Networks Ariel Daliot ∗ , Danny Dolev † and Hanna P arnas ‡ Abstract W e deﬁne the “Pulse Synchron ization” problem that require s nodes to achieve tight synchro nization of regular pulse ev ents, in the settings of d is- tributed computin g systems. Pulse- coupled syn chroniz ation is a pheno menon displayed by a large variety of biolog ical systems, typically overcoming a high level of noise. Inspired by such b iological models, a r obust and self- stabilizing Byzan tine pulse syn chron ization algorithm for distributed com- puter systems is pre sented. T he algor ithm attains near op timal synch roniza- tion ti ghtness while tolerating up to a third of the nodes e xhibiting Byzantine behavior co ncurr ently . Pulse synchronization ha s been previously shown to be a powerful building block for d esigning alg orithms in this se vere fault model. W e hav e p reviously shown how to stabilize general Byzantin e alg o- rithms, using pulse synchron ization. T o the best of our knowledge there is no other scheme to do this without the use of synchro nized pulses. Ke ywords : Self-stab ilizati on, Byzantin e fau lts, D istrib uted algorith ms, Rob ust- ness, Pulse synchr onizat ion, Biological synchron izatio n, Biological oscillato rs. 1 Introd uction The pheno menon of synchroniz ation is display ed by many biological syst ems [ 32 ]. It presumab ly plays an impor tant role in thes e systems. For e xample, the he art of the lobster is re gula rly acti vate d by the synchro nized ﬁring of four in terneu rons in ∗ School of Engineering and Computer Science. The Hebrew Univ ersity of Jerusalem, Israel. Email: adaliot@cs.huji.ac.il † School of Engineering and Computer Science. The Hebrew Univ ersity of Jerusalem, Israel. Email: dolev@cs.hu ji.ac.il ‡ Department of Neurobiology and the Otto Loewi Minerv a Center for Cellular and Molecu- lar Neurobiology , Institute of Li fe S cience, The Hebre w University of Jerusalem, Israel. Email: hanna.parnas @huji.ac.il 1 the cardiac pacemak er network [ 16 , 17 ]. It was concluded th at the org anism cannot survi ve if all fou r int erneur ons ﬁre out of synch ron y for pro longe d times [ 30 ]. This system inspired the presen t work. Other exa mples of biologic al synchr oniza tion includ e the malaccae ﬁ reﬂies in S outheas t A sia where thou sands of male ﬁreﬂies congre gate in mangrov e trees, ﬂashing in synchron y [ 4 ]; o scillat ions of the neurons in the circadian pacemaker , determin ing the day-n ight rhythm; crickets that chirp in unison [ 33 ]; coordinat ed mass spawning in cora ls and ev en audienc e clapp ing togeth er after a “good ” performance [ 28 ]. Synchron izatio n in these systems is typica lly attained despite the in herent v ariati ons among the par ticipa ting elements, or the presenc e of noise from ext ernal source s or from participa ting element s. A generi c mathematica l model for synchro nous ﬁring of biological oscill ators based on a mode l of the human cardiac pa cemak er is gi ve n in [ 27 ]. This model does not accoun t for n oise or for the inherent diff erence s among bio logic al elements . In computer science, synchr onizat ion is both a goal by itself and a bu ildin g block for algorithms that solve other problems. In the “Clock S ynchro nizati on” proble m, it is require d of comput ers to ha ve their clo cks set as close as pos sible to each other as well as to keep a not ion of real-time ([ 11 , 21 , 22 ]). In gene ral, it is desire d for algorithms to gua rantee correct beha vior of t he sys- tem in face of faults or failing elements , without strong as sumption s on the init ial state of the syst em. It has been suggeste d in [ 30 ] that similar f ault consi derati ons may hav e been in vo lve d in the ev olution of distrib uted biological systems. In the exa mple of the cardia c pacemak er netwo rk of the lobst er , it w as concluded that at least four neurons are needed in order to ov ercome the presence of one faulty neu- ron, thou gh sup posed ly one neuro n suf ﬁces t o act i v ate th e hea rt. T he car diac pa ce- maker networ k must b e able to a djust the pace of t he synchro nized ﬁring accordin g to the required heartbeat , up to a certain bound, without losing the synchron y (e.g. while escapi ng a predator a higher heartbe at is required – thou gh not too high) . Due to the vitality of this network , it is presu mably optimized for fault toleranc e, self-st abiliz ation, tight synchro nizati on and for fast re-syn chron ization. The appa rent resemblan ce of the sync hroni zation and fault tolerance require- ments of biolo gical networks and distrib uted computer netwo rks makes it v ery ap- pealin g to infer from models of biological systems onto the design of distrib uted algori thms in c omputer scienc e. Especially whe n assuming th at di strib uted biolog- ical netwo rks ha ve ev olved ov er time to particula rly tolerat e inhere nt h eterog eneity of the c ells, noise and cell d eath. In the current p aper , we sho w that in spite of obvi- ous dif fere nces, a biolog ical f ault tolerant sync hroniz ation model ([ 30 ]) can i nspire a nov el so lution to an apparentl y similar probl em in computer science. W e propos e a relax ed versi on of the Clock Synchron izatio n problem, w hich we call “Puls e Synchroniza tion”, in which all the elements are required to in v ok e some reg ular pulse (or perform a “tas k”) in tight synch ron y , but allo w s to de viate 2 from exact regularit y . Though nodes need to in vok e the pulses synchr onous ly , there is a limit on how fre quentl y it is allo wed to be in v oked (similar to the linear en velo pe clo ck synch roniza tion limitation). The “Pulse Synch roniza tion” prob lem resemble s physical/ biolog ical pulse-cou pled synch roniza tion models [ 27 ], though in a computer system setting an algorit hm needs to be suppl ied for the nodes to reach the synchro nizati on requirement. T o the best of our kno wledge this problem has not been formally deﬁned in the setting s of dis trib uted computer systems. W e pr esent a nov el a lgorit hm in the settings o f s elf-st abilizi ng di strib uted algo- rithms, instr ucting the no des ho w and when to in vok e a pulse in order to meet the synch roniza tion requirement s of “Pulse Synchroniza tion”. The core elements of the algorith m are anal ogous to the neurobiolo gical principle s of endo g enous (self genera ted) period ic spiking , summation and time depend ent r efr acto riness . The basic algorithm is quite simple: e ver y node in vok es a puls e regular ly and sends a message upon in v oking i t ( endo genous perio dic s piking ). The node sums messages recei ved in some “windo w of time” ( summation ) and compares this to the con- tinuou sly decreasing time depend ent ﬁring threshold for in v oking the pulse ( time depen dent r efr acto ry functio n ). The node ﬁres when the counte r of the summed messages cro sses the curren t threshold le vel, an d the n resets its c ycle. For in-depth exp lanati ons of these neur obiol ogical terms see [ 20 ]. The algorit hm perfo rms correctly as long as less than a third of the nodes be- ha v e in a completely arbitrary (“Byzantine” ) manner concur rently . It ensures a tight sync hroniz ation of the puls es of all co rrect node s, while not u sing an y central clock or glob al pu lse. W e assume the communicatio n network allo ws for a broad- cast en vironment and has a bounde d de lay on message transmission. The algorithm may not reach its goal as l ong as these limitatio ns are violated or the network graph is disconnect ed. The algorithm is self-stab ilizin g Byzantin e and thus cop es with a more se ver e f ault model th an the tra dition al Byzantine faul t mode l. Classic Byzan- tine algorithms, which are not design ed with self-st abiliz ation in mind, typical ly make use of assumption s on the initial state of the system such as assuming all clocks are initial ly synchron ized, (c.f. [ 11 ]). Observ e that the system might tem- porari ly be thro wn out of the assumption boundarie s, e.g. when more than one third of the nodes are Byzantine or m essage s of corr ect nodes get lost. When the system e v entual ly returns to beha ve according to these presumed assumpti ons it may be in an arbit rary state. A classic B yzantin e alg orithm, being no n-stab ilizin g, might not recov er from this state. On the other hand, a self-st abiliz ing protocol con ver ges to its goal from any state once the system beha ves w ell again, but is typica lly not resilient to permanent fa ults. For our protocol, once the system com- plies with the theoretic ally requir ed boun d of f < 3 n p ermanent Byzantine f aulty nodes in a network of n nodes then, re gardless o f the state o f the system, tight pulse synch roniza tion is achi e ve d within ﬁnite time. It ov ercomes transien t failure s and 3 permanen t Byzantine faul ts and mak es no assumpti ons on an y initi al synch roniz ed acti vity among the nodes (such a s ha ving a common referenc e to time or a common e ve nt for triggerin g init ializa tion). Our algorithm is un iform, all nodes execu te an identical al gorith m. It does not suf fer fro m communi cation deadlo ck, as can hap pen in messag e-dri ven algorithms ([ 3 ]), since the n odes ha ve a time-dep enden t state cha nge, at the end o f which the y ﬁre endoge nously . The faulty nodes cannot ruin an alread y attained synch roniz a- tion; in the worst case, they can slow down the con ver gence to wards synchron iza- tion and speed up the synch ronize d ﬁring freque ncy up to a certain bound . The con ver gence time i s O ( f ) cyc les with a near optimal s ynchr onizat ion of the p ulses to within d real-time (the bound on the end to end network and process ing delay). W e sho w in Subsection 3.3 how the algori thm can be ex ecuted in a non-br oadca st netwo rk to achie v e synchroniz ation of the pulses to within 3 d real-time. Applica tions and contrib ution of this paper: W e ha ve sho wn in [ 6 ] ho w to stabili ze general Byzantine algorithms using synchron ized pulses . In [ 8 ] we ha ve presen ted a ve ry efﬁcie nt, besides being the ﬁ rst, self-st abilizi ng Byzantine token passin g algo rithm. The efﬁcient self-stabili zing Byzantine clock synchron izatio n algori thm in [ 5 ] is a lso t he ﬁrst such a lgorit hm for clock synchron izatio n. All these algori thms assume a backgrou nd self-s tabili zing Byzantin e pulse synchron izatio n module though th e particula r pulse s ynchr onizat ion procedure pre sente d in [ 5 ] suf- fers from a ﬂaw 1 . The on ly other self -stabi lizing Byzantine pulse synch roniz ation algori thm (besi des the current work), is to the best of our kno wledge, the one in [ 9 ], which is a correcti on to the one in [ 5 ]. In comparison to the curren t pap er , the pulse sync hroni zation algo rithm in [ 9 ] has a much higher message compl exi ty and worse tightnes s, is more complica ted but it con ver ges in O (1) , does not assume broadc ast and sca les better . T he current pape r is simpler , uses much shorte r mes- sages; it has a smaller message comple xity and intro duces nov el and interesti ng elements to distrib uted comput ing. In the Discussio n, in Sectio n 6 , we postu late that our resu lt elu cidate s the fea- sibilit y and adds a solid brick to the motiv ation to search for and to understand biolog ical mechanisms for rob ustness that can be carried ov er to computer sys- tems. 2 Model and Prob lem Deﬁnition The en vironment is a netwo rk of n nodes, out of which f are faulty nodes, that com- municate by exch angin g messag es. The nodes regu larly in vok e “pulses”, ideally 1 The ﬂa w was pointed out by Mahy ar M alekpour from NASA LaRC and Radu Siminiceanu fro m NIA, see [ 25 ] . 4 e ve ry Cycle real-ti me units. The in v ocatio n of the pulse is expre ssed by sendin g a message to all the node s; this is also referre d to as ﬁ ring . W e assume that the message passi ng allo ws for an authenticat ed identi ty of the sen ders. The commu- nicatio n netw ork does not guaran tee an y order o n m essage s amon g dif ferent nodes. Indi vidual nodes ha v e no access to a central clock and there is no external pulse system. The hardware clock rate (referre d to as the physical timers ) of correct nodes has a bounde d drift, ρ, from real-time rate. When the syst em is not coher- ent then there can be an unbound ed number of concurrent B yzantin e f aulty nodes, the turno ve r rate between faulty and non -fa ulty nodes can be arb itraril y lar ge and the communic ation network may behav e arbitrarily . Eventuall y the system settle s do wn in a coherent s tate in which there at most f < 3 n permane nt Byz antine faulty nodes and the communicat ion ne twork deli vers messages within bounded time. D E FI N I T I O N 2.1. A n ode is non-faulty at times that it complies with the following: 1. (B ounde d Drift) O be ys a glob al constant 0 < ρ << 1 (typically ρ ≈ 10 − 6 ), suc h that for every r eal-time interva l [ u, v ] : (1 − ρ )( v − u ) ≤ ‘ph ysical timer’ ( v ) − ‘physica l timer’ ( u ) ≤ (1 + ρ )( v − u ) . 2. (O bedien ce) O pera tes accor ding to th e corr ect pr otocol. 3. (B ounde d Processin g T ime) Pr ocesses any messag e of the corr ect pr otocol within π r eal-time units of arrival time. A node is conside red faulty if it violates any of the abov e conditions. The fau lty nodes can be Byzan tine. A f aulty node may recov er fro m its fau lty beha vior once it resu mes obeyi ng the condit ions of a non-f aulty node. In orde r to kee p the deﬁnitio ns consistent the “correction ” is not immediate but rather takes a certain amount of time during which the non-fa ulty node is still not counted as a correct node, although it supposedly beha ves “corre ctly” 2 . W e later specif y the time-lengt h of contin uous non-fau lty beha vior require d of a recov erin g node to be co nsider ed corr ect . D E FI N I T I O N 2.2. The communicatio n ne twork is n on-faulty at periods that it com- plies with the following : • (Bounded Tr ansmiss ion Delay) Any m essa ge sent or re ceived by a non-faulty node will arri ve at e very non -fault y nod e within δ r eal-t ime units. 2 For e xample, a nod e may reco ver with arbitrary v ariables, which may violate the v ali dity cond i- tion if considered correct immediately . 5 Thus, our communica tion ne twork model is an “e ve ntual bounded-de lay” com- municati on ne twork . Basic deﬁnitions and notations: W e use the follo wing notatio ns though nodes do not need to maintain all of them as v ariab les. • d ≡ δ + π . T hus, when the communica tion networ k is non-fa ulty , d is the upper bound on the elapsed real-time from the sendin g of a message by a non-f ault y nod e until it is receiv ed and proc essed by ev ery correct nod e. • A pulse is an intern al ev ent tar geted to happen in “tight” 3 synch ron y at all correc t nodes. A Cycle is the “ideal” time interv al length between tw o suc - cessi ve pulses that a node in v ok es, as gi ven by the user . The actua l cyc le length , denoted in re gular capt ion, has upper and lo wer bounds as a r esult of fau lty nodes and the physica l clock sk ew . • σ represents th e upper bound on the real-time w indo w within which all cor- rect nodes in vok e a puls e ( tight ness of pu lse sync hr onization ). Our so lution achie ves σ = d. W e assume that Cycle ≫ σ . • φ i ( t ) ∈ R + ∪ {∞} , 0 ≤ i ≤ n, denotes, at real-ti me t, the ela psed real-time since the last pulse in voca tion of p i . It is also denoted as the “ φ of node p i ”. W e occasion ally omit the refere nce to the time in case it is clear out of the conte xt. For a node, p j , that ha s not ﬁred since init ializat ion of the syste m, φ j ≡ ∞ . • cy cl e min and cy cl e max are va lues that deﬁne the bound s on the actual cyc le length during correc t behav ior . W e achie v e cyc le min = n − 2 f n − f · Cycle · (1 − ρ ) ≤ cyc le ≤ Cycle · (1 + ρ ) = cy cl e max . • message_ decay repres ents the maximal real-ti me a non-faul ty no de will keep a message or a refere nce to it, before deleting it 4 . In accorda nce with Deﬁni tion 2.2 , the netw ork model in this paper is such that e ve ry messa ge sent or recei ved by a non- fau lty node arri ves within bounded time, δ , at all non-fau lty nodes. T he algorithm and its res pecti ve proofs are speciﬁed in a stron ger networ k model in which eve ry message recei ved by a non-fa ulty node 3 W e consider c · d, for some small constant c, as tight. 4 The exact elapsed time un til deleting a messages is speciﬁed in the P R U N E procedure in F ig. 2 . 6 arri ves w ithin δ time at all non-f aulty nodes. The subtle differ ence in the latter deﬁnitio n equals the assu mption that e ver y messa ge rece i ve d by a n on-f aulty node, e ve n a message from a Byzantin e node, will eve ntual ly reach ev ery non-fa ulty node. This weaken s the possibilit y for two-f aced beha vior by Byzantine nodes. The algorithm is able to utilize this fact so that if exec uted in such a netwo rk en- vironmen t, then it can attain a very tight, near optimal, pulse synchro nizat ion of d real- time units. W e show in Subs ection 3.3 ho w to ex ecut e in the back groun d a self-st abiliz ing Byzantine reliable-bro adcas t-like primiti ve , which ex ecutes in the netwo rk model of Deﬁnition 2.2 . This primiti ve effecti vely relays ev ery messag e recei ved by a non-f ault y node so that the latte r netw ork model is sa tisﬁed. In su ch a case the algorithm can be execu ted in the network model of Deﬁnition 2.2 and achie ves synchroni zation of the pulses to within 3 d real-time. Note that the protocol parameters n, f and Cycle (as well as the syste m char - acteris tics d and ρ ) are ﬁxed constants an d th us co nside red part o f the incorrupt ible correc t code 5 . Thus we assume that non -fau lty nodes do not hold arbit rary v alues of these consta nts. A reco v ering node shoul d be considere d corr ect only once it h as been con tinu- ously non-f aulty for en ough time to enable it to h a v e decayed ol d messages and to ha v e exchang ed inf ormation with the other nodes through at least a cyc le. D E FI N I T I O N 2.3. A no de i s corr ect fol lowing cycl e max + σ + message_ decay r eal- time of continu ous non -fault y beh avior . D E FI N I T I O N 2.4. The communica tion network is cor r ect foll owing cy cl e max + σ + message_ decay r eal-time of con tinuo us no n-faul ty behav ior . D E FI N I T I O N 2.5. (System Coheren ce) The system is said to be coher ent at times that it complies with the following : 1. (Q uorum) Ther e a r e a t least n − f corr ect nodes, wher e f is th e upper bound on the number of poten tially no n-cor r ect nodes, at steady state . 2. (N etwork Corre ctness ) The communi cation network is corr ect. Hence, if the sy stem i s not coh erent then there can be an u nbou nded number o f concu rrent f aulty nodes; the turnov er rate between the fault y and non-faul ty nodes can be arbitrari ly lar ge and the communicati on networ k may de li v er messages with unbou nded delays, if at all. The system is consi dered coherent, once the commu- nicatio n netwo rk and a sufﬁci ent fraction of the nodes ha ve been non-fau lty for a 5 A system cannot self-stabilize if the entire code space can be perturbed, see [ 15 ]. 7 suf ﬁ ciently long time period for the pre-con dition s for con ver gence of the proto- col to hold. The assumpt ion in this paper , as under lies an y other self-stabi lizing algori thm, is that the syst em ev entu ally becomes coherent . All th e lemmata , theore ms, corollar ies and deﬁnitio ns hold as long as the sys - tem is coheren t. W e now seek to giv e an accura te and formal deﬁnition of the notion of pulse synch roniza tion. W e start by deﬁning a subset of the syste m states, whic h we call pulse_ states , that are determined on ly by the elapse d real-time since each in di vid- ual node in vok ed a pulse (the φ ’ s). W e then ident ify a su bset of the pulse_sta tes in which some set of correct nodes ha ve ”tight“ or ”close“ φ ’ s. W e refer to such a set as a synchr onized set of node s. T o complete the de ﬁnition of sy nchro ny there is a need to addres s the recurring brief time perio d in which a correct node in a syn- chroni zed set of nodes has just ﬁred while others a re about to ﬁre. This is addresse d by adding to the deﬁnitio n nod es whose φ ’ s are almost a Cycle apart. If all correct node s in the system compri se a synchronize d set of nod es then we say t hat t he p ulse_s tate is a sy nch r onized_ pulse_states of the system . The o bjecti ve of the algor ithm is hence to reach a synchro nized _pulse_state of the system and to stay in su ch a s tate. The methodol ogy to prov e that our algor ithm does e xactly this will be to show ﬁ rstly that a synchron ized set of correct nodes stay syn chroni zed. Secondly , we show that such synchro nized sets of correct nodes inces santly join togeth er to form bi gger syn chron ized sets of nod es. T his goes on unt il a synchro- nized set that encompass es all cor rect nodes in the system is formed. • T he pulse_sta te of the system at real-time t is giv en by : pul se _ state ( t ) ≡ ( φ 0 ( t ) , . . . , φ n − 1 ( t )) . • L et G be the set of all poss ible pulse_state s of a syste m. • A set of nodes, S, is c alled synchr onized at real-time t if ∀ p i , p j ∈ S, φ i ( t ) , φ j ( t ) ≤ cy cl e max , and one of the follo wing is tru e: 1. | φ i ( t ) − φ j ( t ) | ≤ σ, or 2. cy cl e min − σ ≤ | φ i ( t ) − φ j ( t ) | ≤ cy cl e max and | φ i ( t − σ ) − φ j ( t − σ ) | ≤ σ. • s ∈ G is a synchro nized_pulse_sta te of the system at real- time t if the set of corre ct nodes is synchroni zed at real-ti me t. 8 D E FI N I T I O N 2.6. The S elf-Stabilizi ng Pu lse Synchr onizatio n Pr oblem Con verg ence: Startin g fr om an arbitr ary system state , the system r eaches a syn- chr onized_pu lse_state after a ﬁnite time. Closur e: If s is a synchr onized_p ulse_state of the system at r eal-time t 0 then ∀ r eal-time t , t ≥ t 0 , 1. pulse_state (t) is a sync hr onized _puls e_state, 2. In the re al-time interva l [ t 0 , t ] every corr ect node w ill in voke at most a single pulse if t − t 0 ≥ cy cl e min and will in voke at least a single pulse if t − t 0 ≥ cy cl e max . The sec ond Closu re co nditi on intends to tightly bound the ef fectiv e pulse in v o- cation f requen cy wit hin a p riori bounds . This is in o rder to de fy an y tri vial solut ion that coul d synchron ize the nod es, bu t be complete ly unusabl e, such as instructing the nodes to in vok e a pulse ev ery σ time units. Note that this is a stronge r re- quiremen t than the “linear en velo pe progr ession rate” typically required by clock synch roniza tion algo rithms, in which it is only r equire d that cl ock time progr ess as a linear functio n of re al-time. 3 The “Pulse Synchr onization” Algorithm W e no w pr esent the B I O - P U L S E - S Y N C H algorit hm that solves the “Pulse Synch ro- nizatio n” problem deﬁned in D eﬁnition 2.6 , inspired by and following a neuro- biolog ical analog. The r efractory function describ es the time depend enc y of the ﬁring threshol d. At threshold lev el 0 the node in vok es a pulse ( ﬁr es ) endogenously . The algori thm use s se veral sub-proc edure s. W ith the help of the S U M M AT I O N pro - cedure , each node sums t he pulses that it learns a bout during a recent time wind o w . If this sum (ca lled the C ounter ) cros ses the cu rrent (time-depend ent) thresh old for ﬁring, then the node will ﬁ re, i.e broad casts its C ounter value at the ﬁring time. The e xact pro perties of the time windo w for summing mess ages is det ermined by the message decay time in the PR U N E procedure (see Fig. 2 ). W e no w sho w in greater detail the elements and proce dures de scribe d abo ve. The ref racto ry function The Cycle is the predeﬁned time a correct node will count on its timer befo re in- v oking an endog enous pulse . The refractory function, R E F ( t ) : t → { 0 ..n + 1 } , determin es at eve ry moment the thresho ld for in vokin g a ne w pulse. The refrac- tory function is determined by the parameter s Cycle n , f , d and ρ. All correct 9 nodes ex ecute the same proto col with the same pa rameters and hav e the same re- fractor y fun ction. The refract ory func tion is shaped as a monotonousl y decrea sing step function comprised of n + 2 steps , RE F ≡ ( R n +1 , R n , ..., R 0 ) , where step R i ∈ R + is the time length on the node’ s timer of threshol d le vel i. The refrac- tory fun ction RE F , starts at thresho ld le vel n + 1 and decre ases with time to ward s thresh old lev el 0 . The time length of each threshold step is formulated in Eq. 1 : R i =                    1 1 − ρ C y cle n − f i = 1 . . . n − f − 1 R 1 − R n +1 − ρ 1 − ρ C y cle f +1 i = n − f . . . n 2 d (1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 i = n + 1 , (1) Subsequ ent to a pulse in v ocati on the refra ctory fu nctio n is re started at R E F = n + 1 . The node will the n commence thresho ld lev el n only after measuring R n +1 time units on its timer . Threshold le vel 0 ( R E F = 0 ) is reached only if exactly Cycle time units ha ve elapsed on a node’ s timer since the last pulse in vo cation , follo wing which thresho ld le vel n + 1 is reached immediately . Hence, by deﬁnition, P n +1 i =1 R i ≡ Cycle . It is pr ov en later in Lemma 4.2 tha t RE F in Eq. 1 is consisten t with this. The spec ial step R n +1 is called the absolute refra ctory period of the cycle. Follo wing the ne urobio logic al ana logue with the same name, this i s the ﬁrst period after a node ﬁres, during w hich its threshold lev el is in practi ce “inﬁnite ly high” ; thus a node can ne ve r ﬁre within its absolute refracto ry per iod. See Fig. 7 for a graphi cal pr esenta tion of the refrac tory function and its ro le in the main algorith m. The message sent when ﬁring The cont ent of a message M p sent by a node p, is the Counter , which represent s the numbe r of messages recei ved withi n a cer tain time windo w (whos e ex act p rop- erties are described in the appendix) that trigge red p to ﬁre. W e use the notati on C ounter p to mark the local Counter at node p and C ounter M p to mark the Counter contai ned in a rec ei v ed message M p sent by node p. 3.1 The S U M M A T I O N proced ur e A full account of the proof of correctness of the SU M M ATI O N procedure is prov ided in the appendi x. The S U M M A T I O N proced ure is ex ecute d upon the arriv al of a ne w messag e. Its purpos e is to decide whether this message is eligible for being counte d. It is comprised of the follo wing sub-proc edure s: 10 Upon arri val of the ne w message , the T I M E L I N E S S proce dure determines if the Counter containe d in the message seems “plausible” ( timely ) w ith respect to the number of other messages receiv ed recen tly (it als o waits a short time for such messages to possibly arriv e). The bound on message transmissio n and process- ing time among correct nodes allo ws a node to estimate whether the content of a message it recei ves is plausi ble and therefo re timely . For examp le, it does not make sense to consi der an arriv ed message that states that it was sent as a resul t of recei ving 2 f m essages , if le ss tha n f messages ha ve bee n rec ei v ed durin g a re- cent time windo w . Such a message is clearly seen as a fa ulty node by all correct nodes . On the other , a message that states that it was sent as a result of receiv- ing 2 f messages, when 2 f − 1 messages ha ve been recei ved duri ng a recent time windo w does not bear enough inf ormation to decide whether it is f ault y or not , as other correct no des may hav e decided that this message is timely , due to rec ei ving a fa ulty messa ge. S uch a message needs to be tempo rarily tabled so that it can be recons idered for being cou nted in case so me correc t nod e se nds a message with in a short time, and which has co unted that faulty message. T hus, intuiti ve ly , a me ssage will be timely if the Coun ter in that message s is less or equal to the total number of ta bled or ti mely mes sages that wer e rec ei ve d within a short recent time win do w . The exac t length of the “recent” time window is a cruc ial factor in the algorith m. There is no ﬁxed time after which a message is too old to be timely . The time for message e xchange betwee n cor rect no des is ne ver d elayed beyon d the netwo rk and proces sing delay . Thus, the ﬁre of a correct node, as a cons equen ce of a messa ge that it recei v ed, adds a bound ed amount of relay time. This is the bas is for the time windo w within which a speciﬁc Counter of a message is checked for plausibil ity . Hence, a particu lar Counter of a message is plaus ible only if there is a suf ﬁcient number of other messages (tabled or not) that were recei ved within a suf ﬁciently small time windo w to hav e been relayed from one to the other within the bound on relaying between correct nodes. As an examp le, consid er that the bound on the allowed relay interva l of m essage s is taken to be 2 d time units. Suppose that a corre ct node recei v es a m essage with Counte r that equals k . T hat m essage will only be c onsid ered as timely i f the re are at least k + 1 messages that were recei ved (inclu ding the last one) in the last k · 2 d time windo w . This is the main criterion for bein g timely . On termination of th e procedur e the message is said to h a ve b een assess ed . If a message is assessed as timely then the M A K E - AC C O U N T A B L E procedure determin es by ho w much to increment the Counter . It does so by conside ring the minimal number of recently tabled messages that were needed in order to assess the message as timely . This number is the amount by w hich the C ounter is incremented by . A tabled messag e is marke d as “uncoun ted” because the node’ s Counter do es not reﬂect th is message. T abled messages that are used for assessi ng a message as 11 timely become mark ed as “counted” because the node’ s Counter now reﬂec t these message as if they were initially timely . A node’ s Counter at ev ery moment is exa ctly the number of messages that are marked as “counte d” at t hat moment. The P R U N E proced ure is responsi ble for the tabling of m essages . A cor - rect node wishes to mark as counted, only those messages which conside ring the elapse d time sin ce their arr i v al, will together pass the criterio n for bein g timely at any correct nodes r ecei ving the conseque nt Counte r to be sent. T hus, messages that were initially assessed as timely are table d after a sh ort while. This is what cau ses the Coun ter to dissipate . After a certain time messages are deleted altog ether ( de- cayed ) . S U M M A T I O N ( a new messag e M p arrived at time t a rr ) / * at node q * / if ( T I M E L I N E S S ( M p , t a rr ) == “ M p is timely”) then M A K E - A C C O U N TA B L E ( M p ) ; / * possibly increment C ounter q * / P R U N E ( t ) ; Figure 1: T he S U M M A T I O N procedure The tar get of the S U M M A T I O N procedur e is formulated in the follo wing two proper ties: Summation Propert ies: Foll o wing the arri val of a message from a correct node: P1: T he message is asses sed within d real-time un its. P2: Follo wing assessment of the messa ge the recei ving node’ s Counter is incre- mented to hold a v alue grea ter than the Counter in the message. The S U M M A T I O N procedure satisﬁes the Summation Properties by the follo wing heuris tics: • When the Counter crosses the threshold lev el, either due to a sufﬁci ent counter incremen t or a thresho ld decrement, then the node sends a message (ﬁres). The message sent holds the v alue of Coun ter at sending time. • T he T I M E L I N E S S procedur e is emplo yed at the recei ving node to assess the credib ility (ti meliness ) of the v alue of th e Counter contained in this m essage . This proced ure ensures that message s sent by correc t nodes with Counter less than n will alw ays be assessed a s timely b y other correct nodes recei ving this message. • When a recei ved message is declar ed timely and therefo re accounted for it is store d in a “count ed” message buf fer (“Counted Set”). The receiv ing 12 node’ s C ounter is then updated to hold a v alue greater than the Counter in the messag e by the M A K E - A C C O U N TA B L E procedure . • If a messag e recei v ed is declar ed untimely then it is te mporaril y stored in an “unco unted” message buf fer (“Uncounted Set”) and will not be accoun ted for at this stage. O ver time, the timeliness test of pre viou sly store d timely messages may no t hol d an y more. In this case, su ch messa ges will be mov ed from the Counted Set to the Uncoun ted Set by the P R U N E procedu re. • A ll messages are deleted aft er a certain time-perio d (messag e dec ay t ime) by the P RU N E proce dure. Deﬁnitions and state vari ables: Counter : an inte ger repre sentin g the node’ s estimation of the number of timely ﬁring e vents re cei ved from di stinct nod es within a certain time win do w . Counter is update d upon recei ving a timely mess age. The node’ s Counte r is che cke d against the refract ory function whene ver one of them changes. The va lue of Counter is bound ed and change s non-monot onous ly; the arri val of timely e vents may inc rease it and the decay /untimel iness of old ev ents may de crease it. Stor ed message : is a basic data structure represe nted as ( S p , t arr ) and created upon arri val of a messa ge M p . S p is the id (or sign ature) of the sendin g no de p a nd t arr is the local arri val time of the m essage . W e sa y tha t two stored messag es, ( S p , t 1 ) and ( S q , t 2 ), are distinct if p 6 = q . Counted Set (CS ) : is a set of distin ct stored messages that determine the cur- rent v alue of Counter . The Counter reﬂects the number of stored messages in the Counted Set. A stored messa ge is acco unted f or in Counter , if it was in CS when the current v alue of Counter was dete rmined. Uncounted Set (U CS) : is a set of stored messages, not necessarily distin ct, that ha v e not been accoun ted for in the current val ue of Counter and that are not yet due to decay . A stored message is placed (tabled) in the U CS when its message clearly reﬂects a faulty sending node (such as when multiple messages from the same node are recei ved) or because it is not timely anymore. Retir ed UCS ( R UCS) : is a set o f dis tinct stored mess ages not ac count ed for in the curren t va lue of Counter due to the elapsed local time since their arriv al. These stored messages are awa iting deletion (decay ing). 13 The CS and UCS are mutually exclus i ve and together reﬂect the messages re- cei ved from oth er nodes in th e precedin g time windo w . T heir union is denoted the node’ s Message _P ool . t s e n d M p : denotes the local-time at whic h a node p sent a message M p . An equi v- alent deﬁniti on of t s end M p is the loc al-time at which a recei ving no de p is ready to assess wheth er to s end a message con sequen t to the ar ri v al and proces sing of some other message. M essag eAg e ( t, q , p ) : is the elapse d time, at time t, on a nod e q ’ s cloc k sin ce the most recent arri v al of a message from node p, which arriv ed at loca l-time t arr . Thus, its va lue at node q at current local-time t is gi v en by t − t arr , where M p is the most rec ent message that a rri ved from p. If no s tored messag e is held at q for p then M ess ag eAg e ( t, q , p ) = ∞ . C S Ag e ( t ) : denotes, at local- time t, the lar gest M essag eAg e ( t, q , . . . ) among the stored messages in CS of node q . τ : denot es the fun ction τ ( k ) ≡ 2 d (1 + ρ ) ( 1+ ρ 1 − ρ ) k +1 − 1 ( 1+ ρ 1 − ρ ) − 1 . The set of proc edure s used by the S U M M A T I O N p ro cedur e (at node q ): The following procedure moves and deletes obsolete stored messages. It prunes the CS to hold only stored messages such that a message sent holding the resultant Counter will be assessed as timely at any correct node receiving the message. P RU N E ( t ) / * at node q * / • De lete from RUCS all entries ( S p , t ) whose M e ssag eAg e ( t, q , p ) > τ ( n + 2); • Mo ve to RUCS, from the Message_Pool , all stored messages ( S p , t ) whose M essa g eA g e ( t , q , p ) > τ ( n + 1); • Mo ve to UCS, from CS, stored messages, beginning with the oldest, until: C S Ag e ( t ) ≤ τ ( k − 1) , where k = max[1 , k C S k ]; • Se t C ounter := k C S k ; Figure 2: T he P R U N E procedu re 14 We say that M p has been assessed by q , once the following procedure is completed. A message M p , is timel y at local time t ar r at node q once it is declared timely by the procedure, i.e. 1: whether the Counter in the message is within its valid range; 2: whether the sending node has recently sent a message, in which only the latest is considered; 3: whether enough messages have been received recently to support the credibility of the Counter in the message. T I M E L I N E S S ( M p , t ar r ) / * at node q * / / * check if Counter is valid * / Timeliness Condition 1: If ( 0 ≤ C ounter M p ≤ n − 1 ) Then Create a new stored message ( S p , t ar r ) and insert it into UCS; Else return “ M p is not timely”; / * if an older message from same node already exists then must be a faulty node. Delete all its entries but the latest. * / Timeliness Condition 2: If ( ∃ ( S p , t ) , s.t . t 6 = t ar r , in Messa ge_Pool ∪ RUCS) Then a delete from Message_Pool all ( S p , t ′ ), where t ′ 6 = t ar r ; return “ M p is not timely”; / * check if C ounter M p seems credible with respect to the Message_Pool * / Timeliness Condition 3: Let k denote C ounter M p . If (at some local-time t in the interval [ t ar r , t ar r + d (1 + ρ )] : k{ ( S r , t ′ ) | ( S r , t ′ ) ∈ Message_ Pool , M essa g eA g e ( t , q , r ) ≤ τ ( k + 1) }k ≥ k + 1 ) Then b return “ M p is timely”; Else return “ M p is not timely”; a W e assume no concomitant messages are stamped with t he exact same arriv al times at a correct node. W e assume that one can uniquely identify messages. b W e assume the implementation can assess t hese conditions within the time windo w . Figure 3: T he T I M E L I N E S S proce dure 15 This procedure moves stored messages from UCS into CS and updates the value of Counter. This is done in case the arrival of a new timely message M p , ha s made previously uncounted stored messages eligible for being counted. M A K E - AC C O U N T A B L E ( M p ) / * at node q * / • Mo ve the max[1 , ( C ounter M p − C ounter q + 1)] mo st recent distinct stored messages from UCS to CS; • Se t C ounter := k C S k ; Figure 4: T he M A K E - AC C O U N TA B L E procedu re This procedure causes the effective cycle of the node to be reset, meaning that the RE F fu nction starts the cycle from the highest threshold level again and down to threshold level 0. C Y C L E - R E S E T () / * at node q * / • Re start RE F at R E F := n + 1; Figure 5: T he C Y C L E - R E S E T procedu re W e no w cite the main theorems of the S U M M A T I O N proced ure. The proofs are gi v en in the appendix . Theor em 1. A ny mess ag e , M p , sent by a corr ect node p will be as sessed as timely by every cor r ect nod e q . Lemma 3.1. F ol lowing the arrival of a timely messa ge M p , at a node q , then at time t s end M q , C ounter q > C oun t er M p . Theor em 2. The S U M M A T I O N pr oced ur e satisﬁ es the Summati on Pr operties. Pr oof. Let p denote a correct node that sends M p . T heorem 1 ensures that M p is assessed as timely at eve ry correct node. Lemma 3.1 ensures that the value of C ounter will not decreas e belo w C oun ter M p + 1 until local-t ime t s end M p , thereby satisfy ing the Summatio n Properties . 16 3.2 The eve nt driven “pulse synchron ization” algorithm Fig. 6 sho w s the main algorithm. Fig. 7 illust rates the mode of operation of the main algorith m. B I O - P U L S E - S Y N C H ( n, f , Cycle ) / * at node q * / • It is as sumed that all the parameters and variables are verified to be within their range of validity. • t is the local -time at the moment of executing the respective statemen t. if ( a new message M p arrives at time t a rr ) then S U M M A T I O N ( ( M p , t a rr ) ); if ( C ounter q ≥ R E F ( t ) ) then Broadcast C ounter q to all nodes; / * invocation of the Pulse * / C Y C L E - R E S E T (); if ( cha nge in threshold level according to RE F ) then P R U N E ( t ); if ( C ounter q ≥ R E F ( t ) ) then Broadcast C ounter q to all nodes; / * invocation of the Pulse * / C Y C L E - R E S E T (); Figure 6: T he e v ent dri v en B I O - P U L S E - S Y N C H algorith m Counter “Pulse” - Message Sending a. end of cycle n+1 b. cycle reset time R e f r a c t o r y   f u n c t i o n t h r e s h o l d Figure 7: Schematic example of the mode of operation of B I O - P U L S E - S Y N C H : (a.) The no de’ s Counter (the summed messages) does not cross the threshold during the cycle, l etting the refractory function reach zero and consequently the node ﬁres en dogeno usly . (b .) Sufﬁcient messages from other nodes are receiv ed in time windo w for the Counter to surpass the current threshold, conse- quently the node ﬁres early and resets its cycle. 17 3.3 A Reliable-Br oadcast Primitive In the current subsection we show that the B I O - P U L S E - S Y N C H algorithm can also operate in networks in w hich Byzantine nodes may exhibit true two-fac ed beha vior . This is done by execut ing in the back groun d a self-stabili zing Byzan- tine reliable-b roadc ast-lik e primiti ve, which as sumes no synchroni city whatsoe ver among the nodes. It has the pro perty of relaying an y message rec ei v ed by a corre ct node. Hence, this pri miti ve satisﬁes the broadca st assumpt ion of Deﬁnitio n 2.2 by supply ing a prop erty similar to the relay prop erty of the reliab le-bro adcas t primi- ti ve in [ 31 ]. That latter pr imiti ve assumes a synch ronou s initializatio n and can th us not be used as a bu ilding block for a self-sta bilizin g algorithm. In [ 7 ] we presented the I N I T I A T O R - A C C E P T primitiv e. W e say that a node does an I -accept of a message m sent by some no de p (denoted h p, m i ) if it accep ts that this message was sen t by node p . The I N I T I A T O R - A C C E P T primitiv e es sentia lly satis ﬁes the follo wing two prop - erties (rephra sed fo r our purposes ): IA-1A ( Corr ectness ) If all cor rect nodes in vok e I N I T I A T O R - A C C E P T h p, m i within d real-time of each other then all correct nodes I-accept h p, m i within 2 d real- time unit s of the time the last correc t node in vok es the primiti ve I N I T I A T O R - A C C E P T h p, m i . IA-3A ( Relay ) If a corre ct node q I-accept s h p, m i at rea l-time t , then ev ery correct node q ′ I-accept s h p, m i , at s ome real-time t ′ , with | t − t ′ | ≤ 2 d. The I N I T I A T O R - A C C E P T primitiv e r equire s a correct n ode not to send two suc- cessi ve messages w ithin less than 6 d real-time of each other . Follo wing the B I O - P U L S E - S Y N C H algorithm (see T imelin ess Condi tion 2, in the T I M E L I N E S S pro- cedure ), non-f aulty nodes cannot ﬁre more th an on ce in e ver y 2 d (1 + ρ ) · n > 6 d real-ti me interv al ev en if the system is not coherent, which thus satisﬁes this re- quiremen t. The use of the I N I T I A T O R - A C C E P T primitiv e in our algorithm is by exec ut- ing it in the backgroun d. When a correct node wishes to send a messag e it does so through the primiti ve, which has certai n conditi ons for I-accept ing a message. Nodes may also I -accept messages that where not sent or receiv ed through the primiti ve, if the cond ition s are satisﬁed. In our algorith m nodes will deli ver mes- sages only after the y hav e bee n I -accepted (also for the node’ s o wn message). From [IA-1A] we get that all messages from correct nodes ar e deli vered with in 3 d rea l- time units subsequen t to sending. Fro m [IA-3A] we ha ve that all messages are deli vered within 2 d real-time units of each other at all correc t nodes , ev en if the sender is f aulty . Thus, w e get tha t the ne w network delay ˜ d = 3 d. H ence, the cost 18 of using the I N I T I A TO R - A C C E P T primiti ve is an added 2 d real-time units to the achie ved pulse synchroniz ation tightnes s which h ence becomes σ = ˜ d = 3 d. 4 Pr oof of Corr ect ness of B I O - P U L S E - S Y N C H In this section w e pro ve Closure and Con ver gence of the B I O - P U L S E - S Y N C H algori thm. In the ﬁ rst subsection, 4.1 , we present ad dition al nota tions th at facil itate the proofs. In the second subsec tion, 4.2 , we prov e Closure and in the third, 4.3 , we prov e Con ver gence . The proof that B I O - P U L S E - S Y N C H satisﬁes the pulse sync hroniz ation prob lem follo ws the step s below: Subsecti on 4.1 introduces some notations and procedur es that are for proof purpo ses only . One suc h proce dure partitio ns the corre ct node s into di sjoint sets of synch ronize d nodes (“synchro nized clust ers”). In Subsectio n 4.2 (Lemma 4.4 ), we prov e that “synchro nized clusters” once formed stay as s ynchr onized sets of nodes, this impl ies that onc e the system is in a synch ronize d_pulse_state it remains as such ( Closur e ). In Subsection 4.3 (Theorem 5 ), we pro ve tha t within a ﬁnite number of cycles, the synchro nized clust ers repeatedly absorb to form ev er large r sync hroni zed sets of nodes, u ntil a synchron ized_ pulse_state of the sy stem is r eached ( Con ver gence ) . Note that the the synchron izatio n tightness , σ , of our algor ithm, equals d. It m ay ease follo wing the proofs by thinkin g of the algorith m in the terms of non-li ner dynamics, tho ugh this is not nec essary for the understand ing of any part of the protocol or its proofs. W e show that the state space can be divide d into a small number of stable ﬁxed points (“synchron ized sets”) such that the state of each indi vidual node is attracted to one of the stable ﬁxed points. W e show that there are always at least two of these ﬁxed points tha t are situated in the basins of attract ion (“absorban ce distance”) of each other . Follo wing the dy namics of these attract ors, we sho w that e v entual ly the states of all nodes settle in a limit c ycle in the basin of one attract or . 4.1 Notations, proce dur es and pr operties used in the pr oofs First node in a synchr onized set of node s S, is a node of the su bset of nod es that “ﬁre ﬁrst” in S that satisﬁes: “First node in S ” =  min { i | i ∈ max { φ i ( t ) | node i ∈ S, φ i ( t ) ≤ σ }} ∃ i ∈ S s.t. φ i ( t ) ≤ σ min { i | i ∈ max { φ i ( t ) | node i ∈ S }} otherwise. 19 Equi v alently , we deﬁne last node : “Last node in S ” =  max { i | i ∈ min { φ i ( t ) | node i ∈ S, φ i ( t ) > σ }} ∃ i ∈ S s.t. φ i ( t ) > σ max { i | i ∈ min { φ i ( t ) | node i ∈ S }} oth erwise . The seco nd cases in both de ﬁnitions serv e to identif y the First and Last nodes in case t falls in-be tween the ﬁre of the nodes of the set. Synchro nized Clusters At a giv en time t the nod es are divide d into dis joint synchr onized cluster s in the follo wing way : 1. A ssign the maximal synchron ized set of nodes at time t as a synchron ized cluste r . In case there are s e ve ral maximal sets choose the s et that is harborin g the ﬁrst node of the uniﬁed set of all thes e maximal sets. 2. A ssign the second maximal s ynchr onize d set of nodes that are n ot pa rt of t he ﬁrst synch ronize d cluste r as a sy nchro nized clu ster . 3. C ontinu e until all nodes are exclusi vely assigned to a synchron ized cluster . The synchro nized cluster harborin g the node with the lar gest (necess arily ﬁnite) φ among all the nodes is design ated C 1 . The rest of the synchron ized cluste rs are enumerat ed in versely to the φ of their ﬁrst node, thus if there are m synch ronize d cluste rs then C m is the synchroniz ed cluster w hose ﬁrst node has the lo west φ (besid es perha ps C 1 ). Note tha t at most one sync hroniz ed cluster may ha ve nodes whose actual φ dif feren ces is large r than σ, as it can contain nod es that hav e just ﬁred and nodes just about to ﬁre. The deﬁnition of C 1 implies tha t at the time the nodes are part itione d into synchron ized cluster s (time t abo v e) it may be the only synch ronize d cluster in such a state. The clus tering is done only for illustrati ve purpo ses of the proof. It does not actual ly affe ct the protocol or the beha vior of the nodes. In the proof we “ass ign” the node s to syn chron ized clusters at some time t . From that time on we con sider the sync hroniz ed clusters as a co nstan t partitioni ng of the nodes into disjoint syn- chroni zed sets of nodes and we follow the dynamics of these sets. Thus, once a node is ex clusi vely assig ned to some synch roniz ed cluste r it will stay a member of that synchronize d cluster . W e aim at showing that ev entual ly all synchroniz ed cluste rs become one synchr onized set of nodes. Once such a clustering is ﬁxated we ig nore nod es that ha ppen to fail and forthcomin g recov erin g nodes. O ur proof is based on the observ ation that e ven tually we reach a time window within w hich the permanent number of non-corre ct nodes at ev ery time is bounded by f and during that windo w the whole system con ver ges. 20 O B S E RV A T I O N 4.1. T he synch r onized clusterin g pr ocedur e assig ns eve ry corr ect node to e xactly one synchr onized set of nod es. O B S E RV A T I O N 4.2. Immediately following the synchr onized clustering pr ocedur e no two distinc t synchr onized cluster s comprise one synchr onized set of nod es. W e use the follo wing deﬁnitions and notati ons: • C i − synchron ized clu ster number i. • n i − cardin ality of C i (i.e. number of correct nodes associated with sync hroni zed cluste r C i ). • c − current number of synchron ized cluste rs in the current state; c ≥ 1 . • dist ( a, b, t ) ≡ | φ a ( t ) − φ b ( t ) | is the distance ( φ differe nce) be tween n odes a and b at real-time t. • φ c i ( t ) − is the φ ( t ) of the ﬁrst node in synchr onized clus ter C i . • dist ( C i , C j , t ) ≡ dist ( φ c i ( t ) , φ c j ( t ) , t ) at rea l-time t. If at real-time t there exists no othe r synchroni zed c luster C r , such that φ c i ( t ) ≥ φ c r ( t ) ≥ φ c j ( t ) , then we say that the synchr onized clu sters C i and C j are adjacent at real-time t. W e say t hat two synchronize d clust ers, C i and C j , hav e absorbed if their union comprise s a synchro nized set of nodes. If a node in C j ﬁres due to a message recei ved from a nod e in C i , then, as will be sho wn in Lemma 4.7 , the ine vitable result is that their two sy nchro nized clust ers absorb . The course of action from the arri val of the mess age at a node in C j until C j has absorbed with C i is referred to as the absorbance of C j by C i . W e refer th rough out the pape r to the ﬁre of a synchr onized cluster in stead of referri ng to the sum of the ﬁres of the i ndi vidual nodes in the syn chron ized cluster . In Lemma 7.8 we pro ve that thes e two notation s are equi val ent. In T heorem 3 we show that we can explicitl y deter mine a threshold valu e, ad ( C i ) , that has the property that if for two synchroniz ed cluster s C i and C j , dist ( C i , C j , t ) ≤ ad ( C i ) then C i absorbs C j . W e will call that va lue the “ absorbance distance ” of C i . D E FI N I T I O N 4.1. T he absorba nce distance , ad ( C i ) , of a synchr onized cluster C i , is ad ( C i ) ≡ f + n i X g = f +1 R g r eal-time units. 21 Pro perties used for t he pro ofs W e identify and pro ve sev eral propertie s; one property of the S U M M A T I O N proced ure (Prope rty 1) and sev eral prop erties of RE F (Properties 2-7). These are later used to pro ve the corre ctness of the algorithm. Pro perty 1: See the Summation Propertie s in Subsection 3.1 . Pro perty 2: R i is a monotonic decreasing function of i, R i ≥ R i +1 , for i = 1 . . . n − 1 . Pro perty 3: R i > 3 d + 2 ρ 1 − ρ 2 P n +1 j =1 R j , for i = 1 . . . n − f − 1 . Pro perty 4: R i > σ (1 − ρ ) + 2 ρ 1+ ρ P n +1 j =1 R j , for i = 1 . . . n. Pro perty 5: R n +1 ≥ 2 d (1 + ρ ) ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 . Pro perty 6: R 1 + · · · + R n +1 = Cycle . Consider an y clu sterin g of n − f correct nodes into c > 1 synchron ized clus- ters, in which j ′ denote s the lar gest synchroniz ed. Thus n j ′ is th e number of nodes in the lar gest synch ronize d cluste r and is less or equal to n − f − 1 . The numbe r of nodes in the second lar gest cluster is less or equal to ⌊ ( n − f ) 2 ⌋ . Pro perty 7: c X j =1 ,j 6 = j ′ f + n j X g = f +1 R g + n j ′ X g =1 R g ≥ 1 1 − ρ Cycle , where c X j =1 n j = n − f . (2) W e require the follo wing restrictio n on the relat ionsh ip between Cycle , d, n and f in order to prov e that Properties 3-4 hold: Restricti on 1: Cycle > d · (1 − ρ 2 )[(1 − ρ )( f + 1) + 2(1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ] 1 − ρ n − f − 3 ρ + ρ 2 . (3) W e no w prov e tha t Properties 2-7 are propertie s of RE F : Lemma 4.1. Pr operties 2-5 ar e pr operties of R E F under Restriction 1. 22 Pr oof. The proof for P roperti es 2 and 5 follows immediately f rom the d eﬁnition of RE F in Eq. 1 . Note that R i > R j , for 1 ≤ i ≤ n − f − 1 an d n − f ≤ j ≤ n. Moreov er , for σ = d, Property 4 is more rest ricti ve than P roperty 3. Hence, for sho wing th at Propertie s 3 and 4 are proper ties of R E F it is suf ﬁcient to sho w that R j (where n − f ≤ j ≤ n ) satis ﬁes P roperty 4: R j = R 1 − R n +1 − ρ 1 − ρ C y cl e f + 1 > σ (1 − ρ ) + 2 ρ 1 + ρ n +1 X j =1 R j ⇒ 1 1 − ρ Cycle n − f − 2 d (1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 − ρ 1 − ρ C y cl e > [ d (1 − ρ ) + 2 ρ 1 + ρ Cycle ]( f + 1) ⇒ 1 1 − ρ Cycle − ρ 1 − ρ ( n − f ) Cycle − 2 ρ 1 + ρ ( n − f ) Cycle > [ d (1 − ρ )( f + 1) + 2 d (1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ]( n − f ) ⇒ [ 1 − ρ ( n − f ) 1 − ρ − 2 ρ 1 + ρ ( n − f )] Cycle > d [(1 − ρ )( f + 1) + 2(1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ]( n − f ) ⇒ [ ( 1 n − f − ρ )(1 + ρ ) − 2 ρ (1 − ρ ) 1 − ρ 2 ] Cycle > d [(1 − ρ )( f + 1) + 2(1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ] ⇒ 1 − ρ n − f − 3 ρ + ρ 2 1 − ρ 2 Cycle > d [(1 − ρ )( f + 1) + 2(1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ] ⇒ Cycle > d · (1 − ρ 2 )[(1 − ρ )( f + 1) + 2(1 + ρ ) · ( 1+ ρ 1 − ρ ) n +3 − 1 ( 1+ ρ 1 − ρ ) − 1 ] 1 − ρ n − f − 3 ρ + ρ 2 . (4) This inequ ality is exactly satisﬁed by Restrictio n 1 and thus Eq. 1 satisﬁes Propertie s 3 and 4. Note that for ρ = 0 , the inequality bec omes Cycle > d · ( f + 1)( n − f ) . Lemma 4.2. Pr operty 6 is a pr operty of RE F . 23 Pr oof. R 1 + · · · + R n +1 = ( R 1 + · · · + R n − f − 1 ) + ( R n − f + · · · + R n ) + R n +1 = ( n − f − 1) · 1 1 − ρ Cycle n − f + ( f + 1) · R 1 − R n +1 − ρ 1 − ρ Cycle f + 1 + R n +1 = 1 1 − ρ Cycle − 1 1 − ρ Cycle n − f + R 1 − R n +1 − ρ 1 − ρ Cycle + R n +1 = Cycle . Lemma 4.3. Pr operty 7 is a pr operty of RE F . Pr oof. W e will prov e that the constraint in Eq. 2 is al way s satisﬁed by the refrac- tory functi on in Eq. 1 . Note that Eq. 2 is a linear equation of the R i v alues of R E F . W e denoted n j ′ to be the number of no des in the larg est synchro nized cluster , follo wing some partiti oning o f the correct nodes into synchron ized cl usters . W e want to ﬁnd what is the l ar gest v alue of i such that R i is a va lue with a non -zero co ef ﬁcient in the li near equati on Eq. 2 . This va lue is determined by either the lar gest possible cluste r , which may be of size n − f − 1 (in case all but one of the correc t node s are in one synchro nized cluster 6 ), or by the second-lar gest possib le cluster , which m ay be of size ⌊ ( n − f ) 2 ⌋ (in case all correct nodes are in two possib ly equally sized synch ronize d clusters). Thus the larges t v alue of i such that R i is a valu e with a non-z ero coef ﬁcient equals max[ f + ⌊ ( n − f ) 2 ⌋ , n − f − 1] = n − f − 1 , for n ≥ 3 f + 1 . Thus, follo wing Eq. 1 , each of these R i v alues equals 1 1 − ρ Cycle n − f . There are exa ctly n − f (not necessarily dif feren t) R i v alues in Eq. 2 . Hence, incor porati ng Eq. 1 into Eq. 2 reduc es Eq. 2 to the line ar equati on: ( n − f ) · R i ≥ 1 1 − ρ Cycle , where 1 ≤ i ≤ n − f − 1 . It re mains to sho w that Eq. 1 satisﬁes this constr aint: ( n − f ) · R i = ( n − f ) · 1 1 − ρ Cycle n − f = 1 1 − ρ Cycle . 4.2 Pr oving the Closure W e no w sho w that a sync hroni zed set of nod es stays synchroni zed. T his also im- plies that the constituent node s of a synchro nized clusters stay as a synch ronize d 6 The case in which the n − f correct nodes are in one synch ronized cluster implies the objecti ve has been reached. 24 set of nodes , as a s ynchr onized clus ter is in particular a synchron ized set of nod es. This pro ves the ﬁrst Closure requ irement of th e “Pulse Synchroniz ation” problem in Deﬁnition 2.6 . Lemma 4.4. A set of corr ect nodes that is a synchr onized set at r eal-time t ′ , r e- mains sync hr onized ∀ t, t ≥ t ′ . Pr oof. Let there be a synchronize d set of nodes at real-time t ′ . From the deﬁni- tion of a synchroni zed set of nodes, this set of nodes will stay synchr onized as long as no node in the set ﬁres. This is because the φ dif feren ce between nodes (in real-time units) does not change as long as none of them ﬁres. W e there- fore turn our attention to the ﬁrst occasio n after t ′ at which a node from the set ﬁres. Let us examine the extr eme case of a synchro nized set consistin g of at least two nodes at the maximal allo wed φ diffe rence; that is to say that at time t ′ , dist ( f ir s t _ node, l ast _ node, t ′ ) = σ. Further assume that the ﬁrst node in the set ﬁres with a Counter= k, (0 ≤ k ≤ n − 1) , at so me time t ≥ t ′ at th e v ery begin ning of its threshold lev el k , and without loss of generality is also the ﬁrst node in the set to ﬁre after time t ′ . W e will show that the rest of the nodes in the set w ill ﬁre within the interv al [ t, t + σ ] and thus remains a synchron ized set . Property 1 ensu res that the last node’ s Counter will read at least k + 1 sub- sequen tly to the arri val and assessment of the ﬁrst node ’ s ﬁre, since its Counter should be at least the ﬁrst node’ s Counter plus 1 . The proof of the lemma will be done by s ho w ing th at ri ght after the a ssess ment of the ﬁrst n ode’ s ﬁre, the l ast node canno t be at a th resho ld hig her than k + 1 and thus will necess arily ﬁre. The proof is div ided into the follo w ing steps: 1. S ho w th at when th e ﬁrst nod e is at thr eshold le ve l k then the last nod e is at thresh old lev el k + 1 or lower . 2. S ho w that i f the ﬁrst nod e ﬁres with a Counter = k then due to Propert y 1 and Step 1 the last node will ﬁre con seque ntly . 3. S ho w that the last node ﬁres within a d real-t ime w indo w of the ﬁrst node, and as a result, the new di stance betwee n the ﬁrst and las t node is less than or equa l to σ. Observ e that the extr eme case considered is a worst cas e since if the lar gest φ dif ference in th e set i s less tha n σ th en t he thresho ld le vel of the last node may only be lower . The same arg ument also holds if the ﬁ rst node ﬁres after its beg innin g of its threshold lev el k. T hus the steps of the proofs also apply to any intermed iate node in the synchr onized set an d thus remains as a synchroni zed set of nod es. 25 Step 1: In this step we aim at calcul ating the amount of time on the last node’ s clock remainin g until it co mmences its thr eshol d lev el k , count ing from the e vent in which the ﬁrst nodes commences its thresho ld le vel k . By sho wing that this remainin g time is less than the length of thresh old le vel k + 1 , as counted on the clock of the last and slowes t node we concl ude that this node must be at most at thresh old lev el k + 1 . The calc ulatio ns are done on the slo w nod e’ s clock . Assume the ﬁrst node to be the fas test permissible node and the last one the slo w est. Hence, when the ﬁrst node’ s th reshol d le vel k commences , 1 1 + ρ n +1 X i = k +1 R i (5) real-ti me units actuall y passed since it last ﬁred. The last node “counte d” this period as: 1 − ρ 1 + ρ n +1 X i = k +1 R i . (6) The last no de has to count on its clock, from the time that the ﬁrst node ﬁred, at most σ (1 − ρ ) local-time units (max. φ dif ference of cor rect nodes i n a synchro - nized set as counte d by th e slo west node), and n +1 X i = k +1 R i (7) in order to reach its own threshol d le ve l k . As a result, the maximum local- time dif fere nce between the time the ﬁrst node starts its threshold lev el k till the last node start s its own threshold lev el k as coun ted by the last node is there fore σ (1 + ρ ) plu s the diff erence Eq. 7 – Eq. 6 , which yields σ (1 − ρ ) + 1 + ρ 1 + ρ n +1 X i = k +1 R i − 1 − ρ 1 + ρ n +1 X i = k +1 R i = σ (1 − ρ ) + 2 ρ 1 + ρ n +1 X i = k +1 R i . (8) Property 4 ensures that R k +1 is greater than Eq. 8 for 0 ≤ k ≤ n − 1; thus when the ﬁ rst node commences thresh old l e ve l k the last node mus t be at a threshold le ve l that is less or equal to k + 1 . Step 2: L et the ﬁrst node ﬁre as a result of its Counter equalling k at time t at thresh old le v el k . In case that the last node recei v es almost immediately the ﬁrst node’ s ﬁre (and t hus inc rements its Coun ter to at least k + 1 foll o wing Proper ty 1), it must be at a thre shold le vel that is less or equal to k + 1 (follo w ing Step 1) and 26 will therefore ﬁre. All the more so if the ﬁrst node’ s ﬁ re is recei ved later , since the thresh old lev el can only decreas e in time befo re a node ﬁres. Step 3: W e no w need to estimate the ne w distance between the ﬁrst and last node in order to sh o w that t hey sti ll compris e a synchro nized set. The last nod e assess es the ﬁrst node’ s ﬁre within d real-time units after the ﬁrst node sent its message (per deﬁnition of d ). This yields a distance of d (1 − ρ ) as seen by the last node, which equals the maximal allo wed real-time distan ce, d (= σ ) , between correct nodes in a synchro nized set a t rea l-time t ′ , and thus t hey stay a synch roniz ed set at time t ′ . Cor ollar y 4.5. (Closur e 1) Lemma 4.4 implies the ﬁrs t Closur e co nditio n. Lemma 4.6. (Closur e 2) As l ong as the system state is in a sync hr onize d_pul se_state then the secon d Closur e condit ion hold s. Pr oof. Due to Lemma 4.4 the ﬁrst node to ﬁre in the synchroniz ed set follo wing its previ ous pulse, may do so only if it recei ves the ﬁre from faul ty nodes or if it ﬁres en dogen ously . This may h appen the e arlies t if it re cei ves the ﬁre fr om e xactl y f distinct faul ty nodes . Thus follo wing Eq. 1 its cycle m ight hav e been sho rtened by at most f · Cycle n − f real-ti me units. Hence, in case the ﬁrst node to ﬁre is also a fast node, it fol lo ws th at cyc le min = Cycle · (1 − ρ ) − f n − f · Cycle · (1 − ρ ) = n − 2 f n − f · Cycle · (1 − ρ ) real-ti me units. A no de may ﬁre at the late st if it ﬁres endogen ously . If in addition it is a slo w node then it follo ws that cy cl e max = Cycle · (1 + ρ ) real-ti me un its. Thus in any real-ti me in terv al that is less or equal to cy cl e min any cor rect node will ﬁ re at most once. In an y real-time interv al that is greater or equal to cy cl e max any corre ct node will ﬁre at least once. This concludes the second closure cond i- tion. 4.3 Pr oving the Con vergenc e The proof of Con verg ence is done through sev eral lemmata. W e begin by pre- sentin g suf ﬁcient conditi ons for two synchroniz ed clusters to absorb . In Subsec- tion 4.3.1 , we sho w that the refracto ry fun ction RE F ensures the continuous exis- tence of a pair of synchroniz ed clusters who se uniﬁed set of nodes is not synchro - nized, b ut are within an absorbanc e dista nce and hence absorb . Thus, iterati vely , all synchro nized clusters will e ventual ly ab sorb to form a uniﬁed synchroni zed set of nodes. 27 Lemma 4.7. (Conditions for Absorbance) Given two synchr onized cluster s, C i pr eceding C j , if: 1. C i ﬁr es w ith Counter= k, at r eal-time t c i _ f ir es , wher e 0 ≤ k ≤ f 2. dist ( C i , C j , t c i _ f ir es ) ≤ 1 1 − ρ P k + n i g = k +1 R g − 2 ρ 1 − ρ 2 P n +1 g = k +1 R g then C i will absorb C j . Pr oof. The proof is div ided into the follo wing steps : 1. (a) If C i ﬁres befor e C j , then C j conseq uentl y ﬁres. (b) Subseq uent to the prev ious step: dist ( C i , C j , .. ) ≤ 3 d. 2. Followin g the pre vious step, w ithin one cycle the constitu ent nodes of the two syn chron ized clu sters comprise a synchroniz ed set of nod es. Step 1a: Let us examine the cas e in whic h C i ﬁres ﬁrst at some real-ti me deno ted t c i _ f ir es , and in the worst case that C j doesn ’ t ﬁre before it recei ves all of C i ’ s ﬁre. All the calculat ions assume that at t c i _ f ir es , φ c i ( t c i _ f ir es ) has still not been reset to 0. S peciﬁcall y , assume that the ﬁrst node in C i ﬁred due to incremen ting its Counter to k (0 ≤ k ≤ f ) at the be ginning of its thre shold lev el k . Follo wing Property 1 and Lemma 7.8 the nod es o f C j incremen t their Counters to k + n i after recei ving the ﬁ re of C i . Additio nally , in the worst case, assume that the ﬁrst node in C j recei ves the ﬁre of C i almost immedia tely . W e will no w sho w that th is ﬁre is recei ved at a threshold lev el ≤ k + n i . W e will calcu late the upper -bou nd on the φ of the ﬁrst node in C j at real-time t c i _ f ir es , and hence deduce the upper -bound on its thresho ld lev el. Assu me the nodes o f C i are f ast and the no des of C j are slo w . Should the n odes o f C j be faste r , then the thresh old lev el may only be lo wer . 28 φ c j ( t c i _ f ir es ) = = φ c i ( t c i _ f ir es ) − [ 1 1 − ρ k + n i X g = k +1 R g − 2 ρ 1 − ρ 2 n +1 X g = k +1 R g ] = 1 1 + ρ n +1 X g = k +1 R g − [ 1 1 − ρ k + n i X g = k +1 R g − 2 ρ 1 − ρ 2 n +1 X g = k +1 R g ] = 1 1 + ρ n +1 X g = k +1 R g − [ 1 1 − ρ k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1 R g ] = 1 1 + ρ n +1 X g = k +1 R g − [( 1 1 + ρ − ( 1 1 + ρ − 1 1 − ρ )) k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1 R g ] = 1 1 + ρ n +1 X g = k +1 R g − [ 1 1 + ρ k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1+ n i R g ] = 1 1 + ρ n +1 X g = k +1 R g − [ 1 1 + ρ n +1 X g = k +1 R g − 1 1 − ρ n +1 X g = k +1+ n i R g ] = 1 1 − ρ n +1 X g = k +1+ n i R g . (9) W e now seek to deduc e the bound on C j ’ s threshold le vel at the time of C i ’ s ﬁre. Thus, follo w ing E q. 9 , at real-time t c i _ f ir es the φ of the ﬁrst node in C j is at most 1 1 − ρ P n +1 g = k +1+ n i R g . W e assume d the worst case in which the constitue nt correc t nodes of C j are slo w , thus the se nodes ha ve counted on th eir timers a t lea st (1 − ρ ) · 1 1 − ρ P n +1 g = k +1+ n i R g = P n +1 g = k +1+ n i R g time units since their last pulse. Hence, the corr ect nodes of C j are at re al-time t c i _ f ir es at most in threshold le vel k + n i . Should k < f or the ﬁre of C i be recei ved at a delay , then this may only cause the threshold le v el at time of assessment of the ﬁre from C i to be equal or e ve n smaller than k + n i . Thus, Lemma 4.4 and Property 1 guarantee that the ﬁrst node in C j will thus ﬁre and tha t the r est o f the n odes in b oth synchron ized clust ers will follo w thei r respecti v e ﬁ rst ones within σ real-t ime units. Step 1b: W e seek to estimate the m aximum distance between the two synchro- nized cluste rs followin g the ﬁ re of C j . The ﬁrst node in C j will ﬁre at the latest upon receiv ing and assessing the message of the last node in C i . More precise ly , 29 ﬁre at the latest d real-time units follo wing the ﬁre of the last node in C i , yield- ing a new dist ( C i , C j , .. ) of at most 2 d real-time uni ts re gard less of the pre viou s dist ( C i , C j , .. ) , n i , k and n j . The last node of C j is at most at a distance of d from the ﬁrst nod e of C j therefo re making the maximal dis tance between th e ﬁrst node of C i and the l ast nod e of C j , at the mo ment it ﬁres , equal 3 d real-time units. Step 2: W e will complete the pro of by sho wing tha t after C i causes C j to ﬁre, the two synchron ized clusters act ually abso rb . W e nee d to sho w th at in the c ycle sub- sequen t to Step 1, the nodes th at cons tituted C i and C j become a sync hroniz ed set. Examine the case in which followin g Step 1, either one of the two synchr onize d cluste rs increment its Counter to k ′ and ﬁres at th e begin ning of thresh old lev el k ′ . W e will observ e the φ of the ﬁrst node to ﬁ re, denoted by φ f ir st _ n ode − 2 nd − cy cle . Fol- lo wing the same ar guments as in S tep 1, all other node s increment their Counters to k ′ + 1 after recei ving this node’ s ﬁre. Conside r that this hap pens at the moment that thi s ﬁrst node in crement ed its Coun ter to k ′ and ﬁred, denote d t 2 nd − cy cle − f ir e . Belo w we compute, using Propert y 3, the lower bo und on the φ of the rest of the nodes at real-ti me t 2 nd − cy cle − f ir e , denote d φ o ther − nodes ( t 2 nd − cy cle − f ir e ) . φ o ther − nodes ( t 2 nd − cy cle − f ir e ) ≥ φ f ir st _ node − 2 nd − cy cl e ( t 2 nd − cy cle − f ir e ) − 3 d = 1 1 + ρ n +1 X g = k ′ +1 R g − 3 d = 1 1 + ρ n +1 X g = k ′ +2 R g + R k ′ +1 − 3 d > 1 1 + ρ n +1 X g = k ′ +2 R g + 2 ρ 1 − ρ 2 n +1 X g =1 R g . (10) In the worst cas e, the rest of the constitu ent nod es that were in C i and C j are slo w nod es and thus, at real-time t 2 nd − cy cle − f ir e , coun ted: (1 − ρ ) · ( 1 1 + ρ n +1 X g = k ′ +2 R g + 2 ρ 1 − ρ 2 n +1 X g =1 R g ) = 1 − ρ 1 + ρ n +1 X g = k ′ +2 R g + 2 ρ 1 + ρ n +1 X g =1 R g = 1 − ρ 1 + ρ n +1 X g = k ′ +2 R g + 2 ρ 1 + ρ n +1 X g = k ′ +2 R g + 2 ρ 1 + ρ k ′ +1 X g =1 R g = n +1 X g = k ′ +2 R g + 2 ρ 1 + ρ k ′ +1 X g =1 R g > n +1 X g = k ′ +2 R g . (11) time units since their last pulse. D ue to P roperty 3 all these correct nodes recei ve the ﬁre and incr ement their Counter s to k ′ + 1 in a th reshol d lev el which is 30 less or eq ual to k ′ + 1 and will ﬁre as well within d real-time units of th e ﬁrst no de in the second cycle . Theor em 3. (Conditions for Absorbance) Given tw o sync hr onized c luster s, C i pr e- ceding C j , if: 1. C i ﬁr es w ith Counter= k, at r eal-time t c i _ f ir es , wher e 0 ≤ k ≤ f , and 2. ∃ t, t pr ev _ c j _ f ir ed ≤ t ≤ t c i _ f ir es , for which dist ( C i , C j , t ) ≤ ad ( C i ) then C i will absorb C j . Pr oof. Denote t pr ev _ c j _ f ir ed the r eal-time at which C j pre viously ﬁred befo re time t c i _ f ir es . Gi ven that at some time t, where t pr ev _ c j _ f ir ed ≤ t ≤ t c i _ f ir es , dist ( C i , C j , t ) ≤ ad ( C i ) , we wish to calc ulate the m aximal pos sible distan ce between the two syn- chroni zed clusters at real-time t c i _ f ir es , the time a t which C i ﬁres with Cou nter= k, where 0 ≤ k ≤ f . Under the abov e assumptio ns, the m aximal poss ible distan ce at real-time t c i _ f ir es is obtained when k = f and when at time t pr ev _ c j _ f ir ed the dista nce between C i and C j was exactly ad ( C i ) , i.e dist ( C i , C j , t pr ev _ c j _ f ir ed ) = ad ( C i ) . The uppe r bound on dist ( C i , C j , t c i _ f ir es ) takes into account that from C ′ i s pre viou s real- time ﬁ ring time, t pr ev _ c i _ f ir ed , and until real-ti me t c i _ f ir es , the nodes of C i were fast and that from real-time t pr ev _ c j _ f ir ed and until t c i _ f ir es , th e nod es of C j were slo w . Thus the bound on dist ( C i , C j , t c i _ f ir es ) beco mes the real-time dif feren ce between these: dist ( C i , C j , t c i _ f ir es ) = φ c i ( t c i _ f ir es ) − φ c j ( t c i _ f ir es ) = 1 1 + ρ n +1 X g = k +1 R g − 1 1 − ρ n +1 X g = k +1+ n i R g = 1 1 + ρ k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1+ n i R g = ( 1 1 + ρ − ( 1 1 + ρ − 1 1 − ρ )) k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1 R g = 1 1 − ρ k + n i X g = k +1 R g + ( 1 1 + ρ − 1 1 − ρ ) n +1 X g = k +1 R g = 1 1 − ρ k + n i X g = k +1 R g − 2 ρ 1 − ρ 2 n +1 X g = k +1 R g . (12) 31 Eq. 12 is the uppe r bou nd on the dist ance bet ween the two synchron ized clus- ters a t real-ti me t c i _ f ir es , thus follo wing Lemma 4.7 , the t wo synchr onized clu sters absorb . 4.3.1 Con verg ence of th e Synchro nized Clusters In the coming subse ction we look at the co rrect nodes as parti tioned into synchr o- nized cl usters (at some spe ciﬁc time). Observ ation 4.2 en sures that no tw o of these synch ronize d cluster s comprise one synchroniz ed set of nodes . The objecti v e of Theorem 4 is to sho w that within ﬁnite time, at least two of these synchroniz ed cluste rs will compri se one synchro nized set of n odes. S peciﬁcally , we sho w that in any state that is not a synchro nized _pulse_state of the system, there are at le ast t wo synch ronize d clusters whose uniﬁed se t of nodes is not a sy nchro nized se t but that are within absorbanc e dis tance of each other , and consequ ently the y abs orb . Thus, e ve ntuall y all sync hroniz ed clust ers will c omprise a synchr onized set of nod es. W e claim that if the following relation ship between R E F and Cycle is sat- isﬁed, then absorb ance (of two synchroniz ed clusters whose uniﬁed set is not a synch ronize d set), is ensured irres pecti ve of t he states of the syn chron ized clusters. Let C j ′ denote the larges t synchroni zed cluste r . The theorem belo w , Theorem 4 , sho w s that for a gi ve n cluste ring of n − f correct nodes into c > 1 synchron ized cluste rs and for n, f , Cycle and RE F that satisfy c X j =1 ,j 6 = j ′ ad ( C j ) + 1 1 − ρ n j ′ X g =1 R g ≥ 1 1 − ρ · Cycle (13) there exist at least two synchronize d clusters , whose uniﬁed set is not a synchro- nized set of nodes, that will ev entually underg o abs orban ce. Note that Eq. 13 is deri ved from Prop erty 7 (Eq. 2 ): Eq. 2 deri ves the followin g equation (since the R g v alues are non- neg ati ve), c X j =1 ,j 6 = j ′ f + n j X g = f +1 R g + 1 1 − ρ n j ′ X g =1 R g ≥ 1 1 − ρ · Cycle . (14) Incorp oratin g the abs orban ce distan ce of Deﬁnition 4.1 into Eq. 14 yield s ex- actly Eq. 13 . W e use Eq. 13 in Theorem 4 inste ad of Eq. 2 (Property 7) for read- ability of the proof. Theor em 4. (Absorbance ) Assume a clustering of n − f corr ect nod es into c > 1 sync hr onize d clusters at rea l-time t 0 . Further assume that Eq. 13 holds for the 32 r esulting clustering . T hen ther e will be at lea st one sync hr onized cluster that will absor b some ot her synchr onized clus ter by r eal-t ime t 0 + 2 · cyc le . Pr oof. Note that followin g the synchron ized cluster procedure , the uniﬁed set of the two synchron ized cluster s that will be shown to absorb, are not necessari ly a synch ronize d set of nodes at time t 0 . Assume without loss of general ity th at C j ′ is the sync hroniz ed cluster with the lar gest number of no des, consequen t to run ning the clustering procedure. E xactly one out of the follo wing two poss ibiliti es takes place at t 0 : 1. ∃ i (1 ≤ i ≤ c ) , such tha t dist ( C i , C ( i +1) (mod c ) , t 0 ) ≤ ad ( C i ) . 2. ∀ i (1 ≤ i ≤ c, i 6 = j ′ ) , dist ( C i , C ( i +1) (mod c ) , t 0 ) > ad ( C i ) . Consider case 1. Follo wing th e pro tocol, C i must ﬁre within Cycle local-t ime units of t 0 . Obse rve the ﬁrst real-time, denot ed t i , at which C i ﬁres sub seque nt to real-ti me t 0 . Assume that k ≥ 0 is the number of distinct inputs that causes the Counter of at least on e node in C i to reach the threshold and ﬁre (n ot counting the ﬁre fr om node s in C i itself) . If k > f then at least on e corr ect node outside of C i caused some node in C i to ﬁre. This correct node must belong to some synchro- nized cluster which is not C i . W e denote this synchroniz ed cluster C x as its identity is irrele vant for the sake of the argumen t. W e assumed tha t at lea st one no de in C i ﬁred due to a node in C x . Follo wing Lemma 4.4 the rest of the n odes in C i . will fol- lo w as well, as a synchro nized cluster is in particular a synchron ized set of nodes. This yields a new dist ( C x , C i , .. ) of at most 3 d. Follo wing the same argu ments as in Step 2 of Lemma 4.7 , C x and C i hence absorb . T herefor e the objecti ve is reache d. Hence assume that k ≤ f and that C i did not absorb with an y preced- ing sync hroniz ed clust er . Thus, the last rea l-time that C ( i +1) (mod c ) ﬁred, deno ted t C i +1 − f ir ed , was before or equal to real-time t 0 , i.e. t C i +1 − f ir ed ≤ t 0 ≤ t i and dist ( C i , C ( i +1) (mod c ) , t 0 ) ≤ ad ( C i ) . B y Theor em 3 , C i will abs orb C ( i +1) (mod c ) . Consider case 2. W e do not assume that dist ( C j ′ , C ( j ′ +1) (mod c ) , t 0 ) > ad ( C j ′ ) . Assume that th ere is no absorba nce until C j ′ ﬁres (otherwise th e claim is pro ven ). Let t j ′ denote the real -time at wh ich t he ﬁrst node in C j ′ ﬁres, a t which φ c j ′ ( t j ′ ) = 0 . There are two poss ibilit ies at t j ′ : 2a. ∃ i (1 ≤ i ≤ c ) , such tha t at t j ′ , dist ( C i , C ( i +1) (mod c ) , t j ′ ) ≤ ad ( C i ) . 2b . ∀ i (1 ≤ i ≤ c, i 6 = j ′ ) , dist ( C i , C ( i +1) (mod c ) , t j ′ ) > ad ( C i ) . Consider case 2a. T his case is equi v alent to case 1. The last real-time that C ( i +1) (mod c ) ﬁred, denoted t C i +1 − f ir ed , was before or equal to real-t ime t j ′ . De- note t i the real-time at which the ﬁrst node of C i ﬁres. Thus, t C i +1 − f ir ed ≤ 33 t j ′ ≤ t i and dist ( C i , C ( i +1) (mod c ) , t 0 ) ≤ ad ( C i ) . By T heorem 3 , C i will ab- sorb C ( i +1) (mod c ) . Consider case 2b . W e wish to calculat e φ c j ′ +1 ( t j ′ ) and from this dedu ce the upper bound on the thresho ld lev el of the ﬁrst node in C ( j ′ +1) (mod c ) at real-time t j ′ . W e ﬁrst want to point out that φ c j ′ +1 ( t j ′ ) > c X j =1 ,j 6 = j ′ ad ( C j ) . (15) This stems from the f act that C j ′ has ju st ﬁred and that C j ′ and C ( j ′ +1) (mod c ) are adjace nt syn chron ized clu sters which implies that ∀ i (1 ≤ i ≤ c, i 6 = j ′ + 1) , φ c j ′ +1 ( t j ′ ) > φ c i ( t j ′ ) . Recall that φ c j ′ ( t j ′ ) = 0 . F rom the case consid ered in 2b we h a ve that ∀ i (1 ≤ i ≤ c, i 6 = j ′ ) , dist ( C i , C ( i +1) (mo d c ) , t j ′ ) > ad ( C i ) . Thus Eq. 15 follo ws. Follo wing Eq. 13 and Eq. 15 we get: φ c j ′ +1 ( t j ′ ) > c X j =1 ,j 6 = j ′ ad ( C j ) ≥ 1 1 − ρ · Cycle − 1 1 − ρ n j ′ X g =1 R g . (16) In the worst case the node s of C ( j ′ +1) (mod c ) are slow . Thus at real-time t j ′ the y hav e measured, from th eir last pulse, at lea st (1 − ρ ) · φ c j ′ +1 ( t j ′ ) = (1 − ρ ) · [ 1 1 − ρ · Cycle − 1 1 − ρ P n j ′ g =1 R g ] = P n +1 g = n j ′ +1 R g local-t ime units. Thus, follo wing Property 1, the ﬁrst node in C ( j ′ +1) (mod c ) recei ves the ﬁre from C j ′ and inc rement its Counter to at least n j ′ in a threshold le vel which is less or equal to n j ′ and will thus ﬁre as well. Follo w ing Lemma 4.4 the rest of the synchronize d cluster will follo w as well. This yields a new dist ( C j ′ , C ( j ′ +1) (mod c ) , .. ) of at m ost 3 d. Follo wing the same ar gument s as in Step 2 of Lemma 4.7 , C j ′ and C ( j ′ +1) (mod c ) hence absorb . Thus at l east tw o syn chron ized clusters will abs orb within 2 · cyc le of t 0 which conclu des the p roof. The follo wing theorem assumes the worst case of n = 3 f + 1 . Theor em 5. (Con ver g ence) W ithin at most 2(2 f + 1) · cyc le r eal-time units the system r eaches a sync hr onized _pulse_state . Pr oof. Assume that n = 3 f + 1 . Thus, the maximal number of synchroniz ed cluste rs is 2 f + 1 , and since follo wing Theore m 4 at least two synchr onized clusters absorb in ev ery two cycles we obta in the bound. 34 5 Analysis of the Algorithm and Comparison to Related Algorithms The protocol operat es in two epochs : In the ﬁrst epoc h there is no limitations on the number of failure s and f aulty no des. In this epoch the syst em might be in any state. In the sec ond epoch ther e are at most f nodes that may beha v e arbi trarily at the same time, from which the p rotoco l may sta rt to con ver ge. Nodes may fa il and reco ve r and nodes that hav e just reco v ered need time to sync hroniz e. T herefor e, we assume that e ven tually we ha ve a wind o w of time within which the turno ver be- tween faulty an d non-faul ty nodes is suf ﬁ ciently low and w ithin which the system ine vitably con ver ges (The orem 4 ). A uthentication and fault ratio: The algorith m does not require the po wer of unfor geab le signature s, only an equi valen ce to an authentic ated channel is re- quired . Note that the shared memory model ([ 13 ]) has an implicit assumpti on that is equiv alent to an aut hentic ated chann el, since a no de “k no ws” the ident ity of the node that wrote to the memor y it reads from. A simila r assumption is also implic it in many m essage passing mode ls by assuming a dire ct link among nei ghbor s, and as a result, a node “kno w s” the identity of the sender of a message it recei ves . Many fundamental problems in distr ib uted networks hav e been pro v en to re- quire 3 f + 1 no des to ov ercome f concurrent Byzantine f aults in order to reach a determin istic soluti on without authentic ation [ 18 , 24 , 11 , 10 ]. W e ha ve not sho w n this rela tions hip to be a necessary requireme nt for sol ving the “Pulse Sync hroniz a- tion” probl em bu t the resu lts for relat ed problems lead us to belie ve that a similar result should exis t for the “Pulse Synchroniza tion” prob lem. There are algorit hms that hav e no lower boun d on the number of nodes require d to handle f Byzant ine fault s, but unfor geable signatu res are required as all the signat ures in the message are va lidate d by the recei ver [ 11 ]. This is costly time- wise, it increase s the message size, and it introduces other limitations , which our algori thm does not hav e. Moreov er , within the self-stabil izing paradigm, using digita l signa tures to counter B yzanti ne nodes expose s the protoco ls to “replay- attack ” w hich might empty its usefuln ess. Con verg ence time: W e ha ve sh o wn in [ 5 ] that self-st abiliz ing Byzantine clock synch roniza tion can be deri v ed from self-st abiliz ing Byzantine pulse synchro niza- tion. Con versely , self-stabiliz ing Byzanti ne clock synchro nizati on can be used to tri vial ly produ ce self-stabil izing Byzantine pulse synchro nizat ion. Thu s the two proble ms are suppos edly equally hard. The only self-stabil izing Byzantine clock synch roniza tion algori thms beside s [ 5 ] are found in [ 13 ]. The randomized self- stabili zing Byzantine clock synchron izatio n algorit hm publi shed there synchr o- nizes in M · 2 2( n − f ) steps, where M is the upper bound on the clock v alues held 35 by indi vidual processors . The algorit hm uses message passing, it allo ws trans ient and perman ent faul ts during con ver gen ce, require s at least 3 f + 1 proce ssors, but utilize s a global pulse system. A n additional algorithm in [ 13 ], does not use a global pulse system and is thus parti ally synch ronou s similar to our model. The con ver gence time of the latter algorithm is O (( n − f ) n 6( n − f ) ) . T his is drasticall y higher than our result, which has a cycle length of O ( f 2 ) · d time units and con- ver ges within 2(2 f + 1) cycles. The con ver genc e time of the only other correct self-st abiliz ing B yzantin e pulse synchron izatio n algorit hm [ 9 ] has a cycle length of O ( f ) · d time units and con ver ges within 6 cycle s. Message and space complexity: The size of each messag e is O ( l og n ) bits. Each correct no de mul ticast s exactl y on e messa ge p er c ycle . This yields a mess age comple xity of at most n messages per cycle. The system’ s message complex- ity to reach synchroniz ation from any arbitra ry state is at most 2 n (2 f + 1) mes- sages per syn chron izatio n from an y arbitra ry ini tial state. The faulty nodes cann ot cause the correct nodes to ﬁ re more message s during a cy cle. Comparati ve ly , the self-st abiliz ing clock synchro nizatio n algorithm in [ 13 ] send s n messages during a pulse and thus has a message comple xity of O ( n ( n − f ) n 6( n − f ) ) . This is sig- niﬁcantly lar ger than our message comple xity irres pecti ve of the time inte rv al be- tween the pulses. The messa ge comple xity of the o nly other c orrect self-sta bilizin g Byzantin e pul se synchro nizati on [ 9 ] equa ls O ( n 3 ) per cycl e. The space comple xity is O ( n ) sinc e the v ariabl es maintai ned by the process ors kee p only a linear number of m essage s recentl y recei ve d and vario us other small range v ariables. The number of possible states of a node is lin ear in n and the node does not need to kee p a conﬁguration table. The message broadcas t assumptions, in which ev ery message , e ve n from a fau lty node, ev entually arri v es at all correct nodes , still lea ve s the faulty nodes with certain po wers of multif aced beha vior since we assume nothing on the or der of arriv al of the messages. Consecuti ve messages recei ved from the same source within a short ti me windo w are ign ored, thus, a fau lty nod e can send two concomi- tant m essage s with dif fering v alue s such that two correct nodes might recei ve and relate to dif ferent v alue s from the same faulty node. Tig htness of synchr onizati on: In the presented algorith m, the in voca tion of the pulses of the nodes will be synchro nized to within the bound on the relay time of messages sent and receiv ed by corre ct nodes . In th e broadcast vers ion, this bound on the relay time equals d real-time units. Note that the lo wer bound on clock synch roniza tion in completely connect ed, faul t-free networks [ 23 ] is d (1 − 1 /n ) . W e ha ve sho wn in Section 3.3 ho w the algorith m can be ex ecute d in n on-br oadca st netwo rks to achie v e a synch roniza tion tightnes s of σ = 3 d. Comparati vely , the clock synchroniza tion algor ithm of [ 11 ] reaches a synchro nizati on tightne ss ty pical of clock sy nchro nizati on algorithms of d (1 + ρ ) + 2 ρ (1 + ρ ) · R, where R is the 36 time between re-sy nchro nizatio ns. The second Byzantine clock synch roniz ation algori thm in [ 13 ] reache s a synchroniz ation tightnes s which is in th e magnitude of ( n − f ) · d ( 1 + ρ ) . This is signi ﬁcantly less tight than our result. T he tightne ss of the self-s tabili zing Byzantine pulse synchro nizati on in [ 9 ] equals 3 d real-time units. Firing frequency bound : The ﬁring frequency upper bound during normal steady -state behav ior is around twice that of the endoge nous ﬁring frequen cy of the nodes. This is becau se cy cl e min ≥ Cycle 2 . This bound is inﬂuenced by the fractio n of fa ulty nodes (the sum of the ﬁ rst f threshold steps relati ve to Cycle ). For n = 3 f + 1 this trans lates to ≈ 1 2 Cycle . Thus, if req uired , the ﬁring fr equen cy bound can be closer to the endogenou s ﬁring fre quenc y of 1 · Cycle if the fraction of fau lty nodes is ass umed to be lo wer . For example, for a fract ion of faul t nodes of f = n 10 , the lower boun d on the cycle length , cy cl e min , becomes ap proximat ely 8 / 9 that of the endog enous c ycle length. cy cl e max = Cycle · (1 + ρ ) real-time un its. 6 Discussion W e dev elop ed and presente d the “P ulse Synchroniz ation” problem in gene ral, and an ef ﬁ cient lin ear -time self-s tabili zing Byzan tine pulse syn chron ization algo rithm, B I O - P U L S E - S Y N C H , as a solution in particu lar . The pulse s ynchr onizat ion problem poses th e nodes with the chall enge of in vo king reg ular e vents synchronou sly . T he system may be in an arb itrary state in whic h there can b e an u nboun ded number of Byzantin e faults. The prob lem requir es the pulses to e ventuall y synchron ize from any initial state once the bound on the permanent number of B yzanti ne failur es is less than a third of the network. The probl em re sembles the clock synchron izatio n proble m though there is n o “v alue” (e.g. clock t ime) to a gree on, r ather an e vent in time. Furthermore, to the best of our knowle dge, the only ef ﬁcient self- stabili zing Byzantin e clock synchro nizatio n algorithm assumes a backg round pulse synchro- nizatio n modu le. The algorithm dev eloped is inspired by and shares properti es with the lobster cardia c pacemak er network ; the network elements (the neuron s) ﬁre in tight syn- chron y within each other , whereas the synchronize d ﬁring pace can va ry , up to a certain ext ent, within a linear en velope of a completely regular ﬁring patte rn. A number of papers hav e recentl y postulate d on the similarity between ele- ments connect ed with biologica l robus tness and design principle s in engineerin g [ 1 , 19 ]. In the curren t paper we ha ve observ ed and understo od the mechanisms for rob ustness in a compreh ensib le and vital biolog ical syst em and sho wn ho w to make speciﬁc use of analog ies of these el ements in di strib uted systems in order to attain hi gh rob ustness in a practical manner . The sa me le vel of rob ustne ss has not 37 been practicall y achie ved earlier in distrib uted systems. W e post ulate that our re- sult elucidates the feasibility and adds a sol id bri ck to the motiv ation to search for and to understa nd biologica l m echani sms for rob ustness that can be carried o ve r to compute r systems. The neural netwo rk simulato r SONN ([ 29 ]) was used in early stages of de- vel oping the algorithm for veriﬁcation of the protocol in the face of probabilis tic fau lts an d random in itial states. It is wo rth noting that the pre vious pulse synchr o- nizatio n procedur e found in [ 5 ] was mechanicall y veriﬁed at N ASA LaRC ([ 25 ]) which greatly faci litated unco vering its ﬂaw . A natural nex t step should thus be to und er go simulation and mech anical veriﬁcation of the cu rrent pro tocol that can mimic a true distrib uted syste m f acing transi ent and Byzantine faults. Refer ences [1] U. Alon, M.G. Surette, N. B arkai, S. Leibler , “ Robustness in bacterial chemotaxis ”, Nature, 397(67 15):16 8-71, Jan 14 1999. [2] G. E. And rews, “ The Theo ry o f P artitions ”, Encyclope dia o f Mathematics and Its Applications, V ol. 2, Add ison-W esley , R eading, MA, 1976. [3] J. Brzezi ` nski, and M. Szychowiak, “ Self-S tabilization in Distrib uted Sy stems - a Short Survey , Foun dations of Comp uting an d Decision Scien ces, V o l. 2 5, no. 1 , 2000. [4] J. Buck, and E. Buck , “ Synchr ono us ﬁr eﬂies ”, Scientiﬁc Am erican, V ol. 2 34, p p. 74-85 , May 19 76. [5] A. Daliot, D. Dole v and H. Parnas, “ Linear T ime Byzantine S elf-Stabilizing Clock Synchr on ization ”, Proc. of 7th In ternation al Conferen ce on Principles of Distributed Systems (OPODIS’0 3 La Martinique , France), December, 2 003. Note: A full r e- vised version appears in http://arxi v .org/abs/cs.DC/060 8096 [6] A. Daliot and D. Dole v , “ Self-sta bilization of Byzantine Pr otoc ols ”, Proc. of the 7th Symposium on Self-Stabilizing Systems (SSS’05 Barcelona), pp. 48-67, 2005. [7] A. Daliot and D. Do lev , “ Self-stabilizing Byzan tine Agr eement ”, Proc. of T wenty - ﬁfth ACM Symp osium o n Principles of Distrib uted Com puting (PODC’06), De n- ver , Colorado , J uly 2006. [8] A. Daliot and D. Dolev , “ Self-Stabilizing Byzan tine T oken Circulation ", T echnica l Report TR2005- 77, School of En gineerin g and Compute r Science, T he Hebrew University of Jerusalem, June 2005. Url: http://leibniz.cs.huji. ac.il/tr/834. pdf [9] A. Daliot and D. Dolev , “ Self-S tabilizing By zantine Pulse S ynchr on ization ", Corn ell ArXiv , Aug. 2005. Url: http://arxiv .org/abs/cs.DC/06 08092 38 [10] D. Do lev , J. Halpern, and H. R. Stro ng, “ On the P ossibility a nd Impossibility of Achieving Clock Synchr onization ”, J. of Comp uter and Systems S cience, V ol. 32 :2, pp. 230-25 0, 198 6. [11] D. Dolev , J. Y . Ha lpern, B. Simons, and R. Strong , “ Dynamic F au lt-T olerant Clock Synchr on ization ”, J. Assoc. Computing Machinery , V ol. 42 , No.1, pp. 14 3-18 5, Jan . 1995. [12] S. Dolev , “ Self-Stabilization , ” The MIT Press, 2 000. [13] S. Dolev and J. L. W elch, “ Self-Stab ilizing Clock Synchr onization in the pr esen ce of Byzantine faults ”, Journal of the A CM, V ol. 51, Issue 5, pp. 780 - 79 9, 2004. [14] S. Dolev and J. L. W elch, “ W ait-fr ee clock synchr on ization ”, Algorith mica, 18(4) :486-5 11, 1997. [15] F . C. Fr eiling and S. Ghosh, “ Cod e Stabilizatio n ”, Proc . of th e 7th Symposium on Self-Stabilizing Systems (SSS’05 Barcelona), pp. 128-13 9, 200 5. [16] WO. Fr iesen, “ Physio logical anatomy and burst pattern in the cardiac ganglion of the sp iny lobster P anulirus interuptus ”, J. Com p. Physiol., V ol. 101, pp. 173-189, 1975. [17] WO. Friesen, “ Synap tic interaction in the cardiac gan glion of the spiny lobster P anu lirus i nteruptus ”, J. Comp. Physiol., V ol. 10 1, pp. 191-20 5, 197 5. [18] M. J. Fischer , N. A. L yn ch and M. Merr itt, “ Easy imp ossibility pr oofs for distrib u ted consensus pr oblems ”, Distributed Computing, V ol. 1, pp. 26-3 9, 1986. [19] H. Kitano , “ Biological Rob ustness ”, Nature, V ol. 5, Nov . 2004. [20] C. K och, “ Biop hysics Of Computation: I nformation Pr ocessing In Sin gle Neur ons ”, Oxford Univ ersity Press; October 31, 2004. [21] L . Lampo rt and P . M. Melliar-Smith, “ Synchr onizing Clocks in th e Presence of F a ults ”, Journal of the A CM, vol. 32(1), pp. 52-78, 1985. [22] B. Lis kov , “ Practical Use of Syn chr onized Clocks in Distrib uted Systems ”, Proceed- ings o f 10 th A CM Sympo sium on the Principles of Distributed Compu ting, 1 991, pp. 1-9. [23] J. Lundelius, and N. L ynch, “ An Up per a nd Lower Bou nd for Cloc k S ynchr on iza- tion , ” Information and Control, V ol. 62, pp. 190- 205, Aug/Sep. 1984. [24] N. L y nch, “ Distrib uted Algorithms ”, Morgan Kaufmann, 1996. [25] M. R. Malekp our, R. Siminicean u, “ Comments on the By zantine Self-S tabilizing Pulse Synchr onizatio n Pr oto col: Co unter examples ”, NASA T echnic al Mem oran- dum N ASA-T M2139 51, Feb-2 006. http://hdl.h andle.ne t/2002/16159 39 [26] R. Math ar and J. Mattfeldt, “ Pulse-cou pled decentral synchr onization ”, SIAM J. Appl. Math, V ol. 56, No. 4, pp . 1094-110 6, Aug. 1996 . [27] R. E. Mirollo and S. H. Strogatz, “ Syn chr onizatio n of pulse-co upled biological os- cillators ”, SIAM J. Appl. Math, V ol. 50, pp. 1645 -166 2, 1990. [28] Z . Nèda, E. Ravasz, Y . Brechet, T . V icsek , and A. L. Barabàsi, “ Self-organizing pr ocess: The sound of many hands clapping ”, Natur e, 403, pp. 849-85 0, 2000 . [29] H. P arnas an d E. Si van, “ SONN - Simulator o f Neur onal Networks ”, Hebrew Uni- versity , Jerusalem, Isra el, 1996. http://www .ls.huji.a c.il/ ∽ parn as/Sonn2/sonn.html. [30] E . Sivan, H. Parnas and D. Dolev , “ F ault tolerance in the c ar diac g anglion of th e lobster ”, Biol. Cybern., V ol. 81, pp . 11-23, 1999. [31] S. T oueg, K. J. Perry , T . K. Srikanth , “ F ast Distributed Agreement ”, SIAM Journa l on Computing, 16(3) :445-4 57, June 1987. [32] S. H. Stro gatz and I. Stew art, “ Coupled Oscillators and Biological Synchr on iza- tion ”, Scientiﬁc American, V ol. 26 9, pp. 102-109, Dec. 1993 [33] T . J. W alker , “ Aco ustic synchr ony: two mechanisms in the snowy tr ee crick et ”, Sci- ence, 166, pp. 891-8 94, 196 9. 7 A ppendix Pro of of corr ectness of the S U M M ATI O N pr ocedur e: Lemma 7.1. F or k ∈ N , k ≥ 0 , τ ( k ) · 1 + ρ 1 − ρ + 2 d (1 + ρ ) = τ ( k + 1) . Pr oof. τ ( k ) · 1 + ρ 1 − ρ + 2 d (1 + ρ ) = [2 d (1 + ρ ) ( 1+ ρ 1 − ρ ) k +1 − 1 ( 1+ ρ 1 − ρ ) − 1 ] · 1 + ρ 1 − ρ + 2 d (1 + ρ ) = [2 d ( 1 + ρ ) k X i =0 ( 1 + ρ 1 − ρ ) i ] · 1 + ρ 1 − ρ + 2 d (1 + ρ ) = [2 d (1+ ρ ) k +1 X i =1 ( 1 + ρ 1 − ρ ) i ] + 2 d (1 + ρ ) = 2 d (1 + ρ ) k +1 X i =0 ( 1 + ρ 1 − ρ ) i = 2 d (1 + ρ ) ( 1+ ρ 1 − ρ ) k +2 − 1 ( 1+ ρ 1 − ρ ) − 1 = τ ( k + 1) . 40 Lemma 7.2. Let a corr ect node q r ecei ve a message M p fr om a corr ect node p at local- time t arr . F or e very one of p ’ s stor ed messa g es ( S r , t ′ ) that is accou nted for in C ount er M p , then at q , fr om some time t in the loca l-time interva l [ t arr , t arr + d (1 + ρ )] and at least until the end of th e interv al: M essag eAg e ( t, q , r ) ≤ τ ( C oun ter M p + 1) . Pr oof. Follo wing the P R U N E procedure at p, the oldes t of its stored messages ac- counte d for in C ounter M p was at most τ ( C ounter M p ) time units old on p ’ s clo ck at th e time it sent M p . This oldest stored messag e could ha ve arri ved at q, δ (1 + ρ ) local-t ime units on q ’ s clock , prior to its arr i v al at p. W ithin t his time p should also ha v e recei ved all the message s accounte d for in M p . Ano ther π (1 + ρ ) local- time units could th en ha ve passed on q ’ s clock until M p was sent. M p could hav e arriv ed at q , δ (1 + ρ ) time units on q ’ s clock after it was sent by p. By this time q would also ha ve recei ved all the messages that are accounted for in M p , irrespecti ve if q had pre vious messag es from the same nodes. Another π (1 + ρ ) time units can then pass on q ’ s clock until all messages are proce ssed. T hus, in the worst case that node p is slo w and node q is fas t and by Lemma 7.1 , for e ver y stored message accoun ted for in C oun t er M p , ∃ t ∈ [ t arr + d (1 + ρ )] , we hav e: M essag eAg e ( t, q , r ) ≤ M ess ag eAg e ( t arr + d (1 + ρ ) , q , r ) ≤ τ ( C ounter M p ) · 1 + ρ 1 − ρ + δ (1 + ρ ) + π (1 + ρ ) + δ (1 + ρ ) + π (1 + ρ ) = τ ( C ounter M p ) · 1 + ρ 1 − ρ + 2 d (1 + ρ ) = τ ( C oun t er M p + 1) . Lemma 7.3. The Counter of a corr ect node cannot exce ed n and a corr ect node will not send a Counter that ex ceeds n − 1 . Pr oof. There can b e at mos t n distinct stored messag es in the CS of a correct node hereby bound ing the Cou nter by n. For a correct node to hav e a Counter that equals exactly n it needs its own stored messa ge to b e in its CS, as a co nsequ ence of a message it sen t. Consider the moment after it sent this message , say before the node ’ s Counter reach ed n, that is accoun ted for in its CS. This message was concomitant to its pulse in voc ation and cycle reset. The node asses ses its own message at most d (1 + ρ ) local-time units after s endin g it thus, follo wing the P RU N E procedure , its o wn sto red message will decay at most τ ( n + 2) + d (1 + ρ ) < τ ( n + 3)) = R n +1 local-t ime units after it was sent. Thus at the moment the no de rea ches thr eshold le vel R n its o wn message will already ha ve decay ed and the C ounter will decrease and will be at 41 most n − 1 , imply ing that any messag e sen t by the node ca n carry a Counter of at most n − 1 . Lemma 7.4. A stor ed messag e, ( S r , t ′ ) , that has been moved to the R UCS of a corr ect node q up to d ( 1 + ρ ) local-time units subseque nt to th e ev ent of sendi ng a messa ge M p by p, (or was moved at an earlier time) cannot ha ve been accounted for in C ounter M p . Pr oof. Assume t hat t he st ored mess age ( S r , t ′ ) was mo ved to the RU C S of node q at a local-time t, d (1+ ρ ) local- time units subsequent to the e ve nt t s end M p at node p, (or it was mov ed at an earlier time). Thus at q at local-time t, M ess ag eAg e ( t, q , r ) > τ ( n +1) . Therefore at node p at local- time t s end M p , M essag eAg e ( t s end M p , p, r ) > τ ( n + 1) − 2 d (1 + ρ ) > τ ( n ) . This is because p could hav e recei v ed the message M r up to d (1 + ρ ) lo cal-ti me units later than q did, and q could ha ve recei ved M p up to d (1 + ρ ) lo cal-ti me un its after it was sent . Follo wing the P R U N E procedur e at p , ( S r , t ”) would ha ve been accoun ted for at the sendin g time of M p only if C ounter M p ≥ n + 1 . Therefore by Lemma 7.3 node p did not account for the stored message of r in C ounter M p . Cor ollar y 7.5. A stor ed m essa ge , ( S r , t ′ ) , that has decayed at a corr ect node q prior to the e vent of sen ding a m essa ge M p by p, cannot have been acc ounte d for in C ounter M p . Pr oof. Corollar y 7.5 is an immediate coro llary of L emma 7.4 . Cor ollar y 7.6. Let a corr ect node q r eceive a messag e M p fr om a corr ect node p at local- time t arr . Then, at q , fr om some time t in the local-t ime interval [ t arr , t arr + d (1 + ρ )] and at least until the end of th e interv al: k Message _Pool k ≥ C ounter M p + 1 . Pr oof. Corollar y 7.6 is an immediate cor ollary of L emma 7.2 a nd Lemma 7.4 . Thus, as a conseque nce to the lemmata, we can say informally , that when the system is coherent all correct nodes relate to the same set of messages sent and recei ved. 7.1 Pr oof of Theor em 1 Recall the statement of Theorem 1 : 42 Any messag e, M p , sent by a corr ect node p will be assessed as timely by ev ery corr ect node q . Pr oof. Let M p be sent by a correct node p, and recei ved by a correct node q at local-t ime t arr. W e sho w that the timelines s con dition s hol d: T imeline ss Conditio n 1: 0 ≤ C ounter M p ≤ n − 1 as implied by Lemma 7.3 and by the fact that the CS canno t hold a negati ve number of stored messages. T imeline ss Condition 2: Follo wing Lemma 7.3 a correct node will not ﬁre dur - ing the absolute refractory per iod. Property 5 therefore implie s that a cor rect node canno t count less than τ ( n + 3) local-t ime units betwee n its cons ecuti ve ﬁrings. A pre vious message from a correct node will therefore be at least τ ( n + 2) loc al- time units o ld at a ny other corr ect nod e bef ore it will recei ve an addition al message from that same node. Follo wing the P RU N E procedure , the former message w ill therefo re ha ve decayed at all correct nodes and therefo re cannot be present in the Message _Pool at th e arri val time of the subs equen t m essage from the same se nder . T imeline ss Condition 3: This timeliness condition val idates C ounter M p . The v al- idatio n criterio n relies on the relatio n imposed at the sending node by the P RU N E proced ure, b etween the M essag eAg e ( t, p, .. ) of its accounted stored messages and its current Counter . By Lemma 7.2 , for all store d messages ( S r , t ′ ) accoun ted for in M p , M essag eAg e ( t, q , r ) ≤ τ ( C oun ter M p + 1) from some l ocal-t ime t ∈ [ t arr , t arr + d (1 + ρ )] and until the end of th e interv al. By Corollary 7.6 , k Message _Pool k ≥ C ounter M p + 1 , fro m so me local-time t ′′ ∈ [ t arr , t arr + d (1 + ρ )] and until th e end of the interv al. W e therefore prov ed that T imeline ss Condition 3 holds for any 0 ≤ k < n at the latest at local-t ime t arr + d (1 + ρ ) . The message M p is there fore assessed as t imely by q . Lemma 7.7. F ollowing the arrival and assess ment of a timely m essa ge M p at node q , the subse quent exe cutio n of the M A K E - A C C O U N T A B L E pr ocedur e yields C ounter q > C oun t er M p . Pr oof. W e ﬁrst sho w that at time t, the time of ex ecutio n of the M A K E - A C C O U N T A B L E proced ure, max[1 , ( C ount er M p − C ounter q + 1)] ≤ k UCS k , ensuring the e xis- tence of a suf ﬁ cient number of stored messages in UC S to be mo ve d to CS. M p is assesse d as timely at q , therefore, by Timeline ss Condition 3 and Lemma 7.4 , at time t, 43 C ounter M p < k Message _Pool k = k CS k + k UCS k = C ounter q + k UCS k = C ounter M p − max[1 , ( C ounter M p − C ounter q + 1)] + 1 + k UCS k ⇒ 0 < − max[1 , ( C ount er M p − C ounter q + 1)] + 1 + k UCS k ⇒ max[1 , ( C ount er M p − C ounter q + 1)] − 1 < k UCS k ⇒ max[1 , ( C ount er M p − C ounter q + 1)] ≤ k UCS k . There are two possibi lities at the instant prior to the e xec ution of the M A K E - AC C O U N T A B L E procedure. At this instant C ounter q = k CS k : 1. C ounter M p ≤ C oun t er q , then max[1 , ( C ounter M p − C ounter q + 1)] = 1 , meaning k CS k will increas e by 1 . 2. C ounter M p > C oun t er q , then k CS k will be C ounter q +max[1 , ( C ounter M p − C ounter q +1)] = C oun ter q + C ounter M p − C ounter q +1 = C ounter M p + 1 . In either case, immediate ly subsequ ent to the ex ecuti on of the proc edure we get: k CS k > C ounter M p and therefo re th e updat ed C ounter q > C oun ter M p . 7.2 Pr oof of Lemma 3.1 Recall the statement of Lemma 3.1 : F ollowing the arrival of a timely messag e M p , at a node q , then at time t s end M q , C ounter q > C oun t er M p . Pr oof. Let t arr denote the local- time of arri v al of M p at q . Recall that t s end M q is the local-time at which q is ready to assess wheth er to send a m essage conseque nt to the arri v al an d processing of M p . In the local-time interv al [ t arr , t s end M q ] at least one P R U N E procedure is ex ecute d at q , the one w hich is trig gered by the arri v al of M p . Foll o wing Lemma 7.7 , C oun t er q > C oun t er M p subseq uent to the ex ecuti on of the M A K E - A C C O U N TA B L E procedure . Note that t arr ≤ t send M q ≤ t arr + d (1 + ρ ) . By Lemma 7.4 all stored messages ac count ed for in C ounter M p will not be mov ed out of the Message _Pool by an y P RU N E procedure exe cuted up to local- time t s end M q , th us, C ount er q must stay with a v alue greater than C ounter M p up to time t s end M q . 44 7.3 Lemma 7.8 Lemma 7.8. Let p, q ∈ C i and r ∈ C j , denote thr ee corr ect nodes belongi ng to two diffe r ent sync hr onized cluster s. F ollowing the arrival and assessmen t of p ’ s and q ’ s ﬁr es, both will be accounted for in the Counter of r . Pr oof. W ithout loss of gener ality , assume that p ﬁres before node q . Follo w ing Lemma 4.4 node q will ﬁre within σ of p ( d (1 + ρ ) on r ’ s clock). Node r will recei ve and assess q ’ s ﬁre at a time t q at most d (1 + ρ ) + d (1 + ρ ) = 2 d (1 + ρ ) after p ﬁred. Summation Property [P2] ensures that r will account for each one after their arri val and assessme nts. Furthermore, M essag eAg e ( t q , q , p ) ≤ 2 d ( 1 + ρ ) = τ (0) and there fore node r did not decay or move M p to R UC S by time t q . Therefo re, M p is still account ed fo r by node r at time t q and thus, both p and q are accounted for in C ounter r at time t q . 45

Self-Stabilizing Pulse Synchronization Inspired by Biological Pacemaker Networks

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment