Consensus and synchronization in discrete-time networks of multi-agents with stochastically switching topologies and time delays

Manuscript submitted to W ebsite: http://AIMsciences.org AIMS’ Journals V olume X , Number 0X , XX 200X pp. X–XX CONSENSUS AND SYNCHRONIZA TION IN DISCRETE-TIM E NETWORKS OF MUL TI-A GENTS WITH STOCHASTICALL Y SWITCHING TOPOLOGIES AND TIME DELA YS W E N L I A N L U Center for Computa tional Systems Biology , School of Mathematic al Sciences, Fudan Uni versi ty , Shanghai, China and Max Planck Institut e for Mathematic s in the Sciences, Inselstr . 22, 04103 Leipzig , Germany F A T I H C A N M . A TA Y Max Planck Institute for Mathemat ics in the Sciences, Inselstr . 22, 04103 Leipzig , Germany J ¨ U R G E N J O S T Max Planck Institute for Mathemat ics in the Sciences, Inselstr . 22, 04103 Leipzig , Germany and Santa Fe Institute for the Science s of Complexi ty 1399 Hyde Pa rk Road, Santa Fe, NM 87501, USA (Commun icated b y the as sociate editor name) A B S T R A C T . W e analyze stabili ty of consensus algorithms in networks of multi-age nts with time-v arying topologies and delays. The topology and delays are modeled as induce d by an adapt ed process and are ra ther gene ral, incl uding i.i.d. topol ogy processes, async hro- nous consensus algorit hms, and Marko vian jumping switchi ng. In case the self-links are instant aneous, we prov e that the network reache s consensus for all bounded delays if the graph corresponding to the condition al expectat ion of the couplin g matrix sum across a finite time interv al has a spanning tree almost surely . Moreov er , when self-links are also delaye d and when the delays satisfy certai n integer pattern s, we observe and prove t hat the algorit hm may not reach consensus but instead synchronize at a periodic traject ory , whose period depend s on th e delay pa ttern. W e also gi ve a brie f discu ssion on the d ynamics in the absence of self-link s. 1. Introduction. Consensus pr oblems have been recog nized a s impor tant in distribution coordin ation of d ynamic agent systems, which is widely applied in d istrib uted co mputing [ 21 ], manage ment scien ce [ 5 ], flocking/swarming th eory [ 32 ], distributed control [ 10 ], and sensor networks [ 26 ]. In th ese application s, the multi-agen t systems need to agree o n a common value for a cer tain quantity of inter est tha t depends on th e states of th e in terests of all agents or is a preassigned value. The interaction ru le f or each ag ent specifying the informa tion co mmunication be tween itself and its neighbo rhood is called the con sensus pr otoc ol/algorithm . A related concept of consensus, namely synchr o nization , is considered as “coherence of dif ferent processes”, and is a widely existing phenomeno n in ph ysics and 2000 Mathematics Subject Classificati on. Primary: 93C05, 37H10, 15A51, 40A20; Secondary: 05C50, 60J10. K e y wor ds a nd phr ases. Consensus, sync hroniza tion, delay , netw ork of mult i-agent s, adapt ed proc ess, swit ch- ing topolo gy . 1 2 W . LU, F . M. A T A Y AND J. JOST biology . Synchron ization of interacting systems has been on e of the f ocal po ints in many research and application fields [ 33 , 16 , 29 ]. For more details on consensus and the relation between consensus and synchro nization, the read er is referred to the survey paper [ 27 ] and the references therein. A basic idea to solve the co nsensus pro blem is upd ating the cu rrent state o f each agent by averaging the previous states of its neighb orhood and its o wn. The q uestion then is whether o r under whic h circumstances the multi-agen t system can reach c onsensus by th e propo sed algorithm. In th e past decade, the stab ility analy sis of c onsensus algor ithms has attracted much atten tion in control theor y and mathematics [ 27 ]. The core p urpose of stability analysis is not only to obtain the algebraic conditions for consensus, but also to g et the consensus prop erties of the topology of t he network. The basic discrete-time consensus algorithm can be formulated as follows: x t +1 i = x t i + ǫ X j ∈N i ( x t j − x t i ) , i = 1 , . . . , m, (1) where x t i ∈ R denotes the state variable of th e agent i , t is the discrete-time, N i denotes the neighb orhood o f the agent i , and ǫ is the coupling s trength. Define L = [ l ij ] m i,j =1 as the Laplacian of the grap h of the network in the manner that l ij = 1 if i 6 = j a nd a link from j to i exists, l ij = 0 if that i 6 = j an d no lin k from j to i exists, an d l ii = − P j 6 = i l ij . W ith G = I − ǫ L , ( 1 ) can be re written as x t +1 = Gx t , (2) where x t = [ x t 1 , . . . , x t m ] ⊤ . If the diagona l elements in G are non negati ve, i. e., 0 ≤ ǫ ≤ 1 / max i l ii , then G is a stochastic matr ix. Eq . ( 2 ) is a general mod el o f the synchro nous consensus algo rithm o n a network with fixed topolo gy . The network ca n be a directed graph, for example, the leader-follower structure [ 22 ], and may have weights. In many r eal-world applicatio ns, the co nnection structu re may chang e in time, fo r in- stance when the agents are moving in physical space. One must th en consider time-varying topolog ies u nder link failure or creation. The asynchro nous consensus algorithm also indi- cates that the updating rule varies in time [ 9 ]. Thu s, the consensus algorithm become s x t +1 = G ( t ) x t , (3) where the tim e-v arying co upling matrix G ( t ) expr esses to the time-varying topo logy . W e associate G ( t ) w ith a directed graph at time t (see Sec. 2), in which G ij ( t ) > 0 imp lies that the re is a link fr om j to i at time t , wh ich may b e a self-link if i = j . Note th at the self links in G arise f rom th e presen ce of the x i on the right hand side of ( 1 ); they do not necessarily mean that the physical network of multi-agents ha ve self-loop s. Furthermo re, delays occur inevitably due to limited in formation tran smission sp eed. The consensus algorithm with transmission delays can be described as x t +1 i = m X j =1 G ij ( t ) x t − τ t ij j , (4) where τ t ij ∈ N , i, j = 1 , . . . , m , denotes the time-depend ent delay f rom vertex j to i . A link from j to i is called instanta neous if τ t ij = 0 ∀ t , an d delayed otherwise. In this p aper , we study a general consensu s prob lem in network s with time- v a rying topolog ies an d time delays described by x t +1 i = m X j =1 G ij ( σ t ) x t − τ ij ( σ t ) j , i = 1 , . . . , m, (5) CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 3 as well as the more general form x t +1 i = τ M X τ =0 m X j =1 G τ ij ( σ t ) x t − τ j , i = 1 , . . . , m. (6) Note that ( 5 ) can be put into the form ( 6 ) by partitioning the inter-links acco rding to delays, where τ M is the maximum delay . Howe ver, ( 6 ) is m ore general, as it in principle allows for multiple links with different delay s between the same pair of vertices. In particular, ther e may exist both instantaneous and d elayed self-links, which m ay naturally ar ise in a mo del like ( 1 ) where the term x i appears both by itself as well as und er the su mmation sign . In referenc e to ( 6 ), we talk abou t self-link(s) when G τ ii 6 = 0 , which may b e instanta neous or delay ed dep ending o n whether τ = 0 or τ > 0 , r especti vely . In e quations ( 5 )–( 6 ), σ t denotes a stochastic process, G ( σ t ) = [ G ij ( σ t )] n i,j =1 = [ P τ M τ =0 G τ ij ( σ t )] n i,j =1 is a st ochas- tic matrix , τ ij ( σ t ) ∈ N is the stochastically- v arying transmission delay fro m agent j to agent i . This model can describe, for instance, commu nications between random ly moving agents, wher e the curr ent locatio ns of the agen ts, and hence the links between them, a re regarded as stoc hastic. Furtherm ore, the delay s are also stoch astic sinc e th e y arise due to the d istances between agents. In this paper, { σ t } is assume d to be an ada pted stochastic process. Definition 1. 1. ( A dapted pr o cess ) Let { A k } be a stochastic process d efined on the basic probab ility space { Ω , F , P } , with the state spac e Ω , the σ -alge bra F , and the pro bability P . Let {F k } be a filtration, i.e., a sequence of nond ecreasing sub- σ -algebras of F . If A k is measurable with respect to (w .r .t.) F k , then the sequ ence { A k , F k } is called an adapted process. V ia a standard tr ansformation, any stochastic process can be regarded as an adapted process. Let { ξ t } be a stochastic proce ss in pro bability spaces { Ω t , H t , P t } . Define Ω = Q t Ω t , F and P are both induced by Q t H t and Q t P t , where Q stands for the Cartesian produ ct. L et σ t = [ ξ k ] t k =1 and F t be the minimal σ -algebra induced by Q t k =1 H t . Then F t is a filtration. Thus, it is clea r that th e notion of an ad apted p rocess is rather gene ral, and it contains i.i.d. pro cesses, Marko v chains, and so on, as special cases. Related work. M any r ecent paper s address th e stability analysis of consensus in net- works of multi- agents. Howe ver , the mo del ( 5 ) w ith delay s we h a ve pr oposed above is more g eneral than the existing models in the literature. W e first mention some papers where models of the for m ( 3 ) ar e treated . A result from [ 25 ] shows th at ( 3 ) can reach co nsensus unifor mly if and only if ther e exists T > 0 such that the unio n graph acr oss any T -leng th time period has a spanning tree. Ref. [ 2 ] der i ved a similar condition for reaching a consen- sus via an equivalent concept: str o ngly r ooted gr aph. Our previous papers [ 19 , 20 ] studied synchro nization of nonlin ear dynamical systems of networks with time-varying topologies by a similar method. Ref. [ 36 ] has pointed out that under the assumption that self-links al- ways exist and are instantan eous (i.e. without delays), the co ndition presented in Ref. [ 25 ] also gu arantees co nsensus with arbitrary bounded multiple delays. However , this crite- rion may not work when the time- v a rying topology inv olves ran domness, because f or any T > 0 , it might occu r with positi ve probability that th e union graph acro ss some T -le ngth time period d oes not have a sp anning tree for any T . Refs. [ 14 , 35 , 3 1 ] studied the con- sensus in networks under the circumstance that the processes { G ( t ) } t ≥ 0 are independen tly and identically distributed (i.i.d .) and [ 38 ] also investigated the stability of consensus of multi-agen t systems with Markovian switching topology with finite states. In these papers, consensus is c onsidered in the almo st sur e sense. Ref. [ 8 ] studied a p articular situation with packet drop communication. The most related literature to the cu rrent paper is [ 18 ], where 4 W . LU, F . M. A T A Y AND J. JOST a genera l stochastic pr ocess, an adap ted pr ocess, was introdu ced to mo del the switching topolog y , which genera lized the existing works in cluding i.i.d. and Mar ko vian jump ing topolog ies as special cases. The authors proved that, if the δ -gra ph (see its definition in Sec. 2. 2) correspon ding to the co nditional expectation of the coup ling matrix sum across a finite time inter v al has a spannin g tree almost sur ely , then the system r eaches consensus. Howe ver, n one of those work s consider ed the stoc hastic d elays but rath er assumed that self-links always exist. Th ere are also m any pap ers co ncerned with the co ntinuous-time consensus algorith m on networks of agents with time-varying topolog ies or delays. See Ref. [ 28 ] for a fram e work and Ref. [ 27 ] fo r a survey , as well as Refs. [ 24 , 1 , 37 , 23 ], among others. Also, th ere are paper s concerned with nonline ar coupling functions [ 6 ] and general coordin ation [ 17 ]. Statement of contributions. In the following sectio ns, we study the stability of the consensus of the d elayed system ( 5 ), where σ t is an adapted process. First, we consider the case that each agent contains an instantaneous self-link. In this case, we sho w th at the same condition s enab ling the consensus of algorithms withou t transm iss ion delays, as mentioned in Ref. [ 18 ], ca n also g uarantee co nsensus f or the case of arbitrary boun ded delays. Second , in case th at delays also occur at the self-links (fo r example, wh en it costs time f or e ach ag ent to process its own inf ormation), and only certain delay patterns can occur, we sho w th at the algorithm do es not necessarily reach co nsensus but may synchronize to a periodic trajectory instead. As we show , the period of th e synchron ized state depen ds on the possible delay patterns. Finally , we briefly study th e situation without self-lin ks, and present consensus condition s b ased on the graph topology and the product of coupling matrices. The basic tools we use are theorems about product of stochastic matrices and the results from probab ility theory . Ref. [ 3 ] has proved a necessary and sufficient co ndition fo r the conv ergence o f infinite stochastic m atrix p roducts, which in volv es the conce pt of scram- blingness. Ref. [ 34 ] pr ovided a m eans to get scrambling matrices (defined in Sec. 2 .2) from produ cts of finite stochastic in decomposable ap eriodic (SIA) ma trices a nd Ref. [ 36 ] showed that an SIA matrix can be guarantee d if the correspo nding g raph has a spann ing tree and on e of the roo ts has a self-link. T he Borel-Cantelli lemma [ 7 ] indicates that if the condition al probability of the occu rrence of SIA m atrices in a product of sto chastic matri- ces is always positive, then it occ urs infinitely often. These previous r esults give a bridge connectin g the pro perties o f stoc hastic matrices, graph topolo gies, and probability theory which we will call upon in the present paper . The paper is organized as follows. Intro ductory no tations, d efinitions, an d lem mas are giv en in Sec. 2. The dyn amics of the co nsensus algorith ms in networks of multi-a gents with switching topologies and delays, which are modeled as adapted processes, are studied in Sec. 3. App lications of the results are provided in Sec. 4 to i. i.d. and Markovian jum ping switching. Pro ofs of theorem are presented in Sec. 5. Conclusions are drawn in Sec. 6. 2. Preliminaries. This paper is written in t erms of stochastic process and algebraic graph theory . For the reader’ s con venience, we present some necessary notations, definitions and lemmas in this section. In what follows, N denotes the in te gers fr om 1 to N , i.e., N = { 1 , . . . , N } . For a vector v = [ v 1 , . . . , v n ] ⊤ ∈ R n , k v k deno tes som e norm to be specified, for instance, th e L 1 norm k v k 1 = P n i =1 | v i | . N d enotes the set of positive integers and Z denotes the integers. For tw o integers i and j , we denote by h i i j the quotient integer set { k j + i : k ∈ Z } . The g reatest co mmon divisor of the integers i 1 , . . . , i K is denoted gcd ( i 1 , . . . , i K ) . The produ ct Q n k =1 B k of matrices de notes the left matrix pr oduct B n × · · · × B 1 . F or a matrix A , A ij or [ A ] ij denotes the entry o f A on the i th row and j th co lumn. In a block matrix B , B ij or [ B ] ij can also stand fo r its i , j -th block. For CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 5 two matrices A , B of th e sam e d imension, A ≥ B means A ij ≥ B ij for all i , j , and the relations A > B , A < B , and A ≤ B are defined s imilarly . I m denotes the identity matrix of dimension m . 2.1. Probability theory. { Ω , F , P } is our g eneral notation for a p robability space, which may be different in different contexts. In th is n otation, Ω stan ds f or th e state spa ce, F the Borel σ - algebra, an d P { ·} the probab ility on Ω . E P {·} is the expectation with respect to P (sometimes E for simplicity , if no ambiguity ar ises). For any σ -algebra G ⊆ F , E {·|G } ( P {·|G } ) is the conditio nal expectation (probab ility , respecti vely) with respect to G . It should b e no ted that both E {·|G } an d P {·|G } are actually r andom variables measurable w . r .t. G . The following lemma provid es the general stateme nt of th e principle of large number s. Lemma 2.1. [ 7 ] ( The Sec ond Bor el-Cantelli Lemma) Let F n , n ≥ 0 be a filtration with F 0 = {∅ , Ω } a nd C n , n ≥ 1 a sequ ence of events with C n ∈ F n . Then { C n infinitely often } =  + ∞ X n =1 P { C n |F n − 1 } = + ∞  with a pr oba bility 1 , where ”infi nitely often” means that an infinite number of { C n } ∞ n =1 occur . 2.2. Stochastic matrices and graphs. An m × m matrix A = [ a ij ] m i,j =1 is said to be a stochastic matrix if a ij ≥ 0 for all i, j = 1 , . . . , m and P m j =1 a ij = 1 for all i = 1 , . . . , m . A matrix A ∈ R m,m is said to be SIA if A is stochastic, indecomposable, and aperiodic , i.e., lim n →∞ A n conv erges to a matrix with identical rows. The Hajn al diameter is introduced in Ref. [ 12 , 13 ] to describe the compression rate of a stochastic m atrix. For a matrix A with row vectors a 1 , . . . , a m and a vector norm k · k in R m , the Hajnal diameter of A is defin ed by dia m( A ) = max i,j k a i − a j k . The scramblingness η of a stochastic matrix A is defined as η ( A ) = min i,j k a i ∧ a j k 1 , (7) where a i ∧ a j = [min( a i 1 , a j 1 ) , . . . , min( a im , a j m )] . The stochastic matrix A is said to be scrambling if η ( A ) > 0 . The Hajnal ine quality e stimates the Hajnal d iameter o f the produ ct o f stochastic matrices. For two s tochastic matrices A and B of the same o rder , the inequality diam( AB ) ≤ (1 − η ( A ))diam( B ) (8) holds for any matrix norm [ 3 0 ]. It can be seen fro m ( 8 ) that the diameter of th e pro duct AB is strictly less than that of B if A is scram bling. The link between stochastic matrices and g raphs is an essential feature of this pap er . A stochastic (or simply nonnegative) m atrix A = [ a ij ] m i,j =1 ∈ R m,m defines a graph G = {V , E } , where V = { 1 , . . . , m } denotes the vertex set with m vertices and E de notes the link set wher e there exists a d irected link from vertex j to i , i.e., e ( i , j ) exists, if an d only if a ij > 0 . W e denote this graph corre sponding to the stochastic matrix A by G ( A ) . For a directe d link e ( i, j ) , we say that j is the start of the link and i is th e end o f the link. Th e vertex i is said to b e self-linked if e ( i, i ) exists, i.e., a ii > 0 . G is said to b e a bigraph if the existences of e ( i, j ) and e ( j, i ) are equiv alent. Otherwise, G is said to a digraph . An L - length pa th in th e g raph denotes a vertex sequen ce ( v i ) L i =1 satisfying that the link e ( v i +1 , v i ) exists for all i = 1 , . . . , L − 1 . The vertex i c an acc ess the vertex j , o r equiv alently , the vertex j is accessible from the vertex i , if there exists a path from 6 W . LU, F . M. A T A Y AND J. JOST the vertex i to j . Th e grap h G ha s a spa nning tree if there exists a vertex i wh ich can access a ll o ther vertices, and th e set of vertices that can acc ess all oth er vertices is nam ed the r oot set . The g raph G is said to b e str on gly co nnected if each vertex is a roo t. W e refer inter ested readers to the bo ok [ 11 ] for mor e details. Due to th e relation ship between nonnegative matrices and graphs, we can call on the prope rties of nonnegative m atrices, or equiv alently , those of their correspo nding g raphs. For example, the indecomp osability of a nonnegative matrix A is equiv alent to that G ( A ) has a spanning tree, and the aperiodicity o f a graph is associated with th e aperiodicity of its correspon ding matrix [ 15 ]. W e say that G is scr ambling if for each pair of vertices i 6 = j , ther e exists a vertex k such that both e ( i, k ) and e ( j, k ) exist, which can be seen to be equiv alent to the defin ition o f scr amblingness for stochastic m atrices. For two matrices A = [ a ij ] n i,j =1 , B = [ b ij ] n i,j =1 ∈ R n,n , we say A is an analog o f B and write A ≈ B , in case that a ij 6 = 0 if and only if b ij 6 = 0 , ∀ i, j = 1 , . . . , n , that is, when the ir corresponding graphs are identical. Furthermo re, for a nonn e gativ e matrix A and a given δ > 0 , the δ -matrix of A , deno ted by A δ , is defined as [ A δ ] ij =  δ, if A ij ≥ δ ; 0 , if A ij < δ. The δ -graph o f A is the directed gr aph correspond ing to th e δ -matrix of A . W e de note by N δ i the neighbo rhood set of the verte x v i in the δ -graph: N δ i = { v j : A ij ≥ δ } . 2.3. Conv ergence o f products of s tochastic matrices. Here, we provide the definition of consensus and synchronization of the system ( 5 ). Supp ose the delays are boun ded, nam ely , τ ij ( σ k ) ≤ τ M for all i, j = 1 , . . . , m an d σ k ∈ Ω . Definition 2 .2. The multi-ag ent system is said to reach consensus v ia the algor ithm ( 5 ) if for any essentially boun ded r andom initial data x 0 τ ∈ R m , τ = 0 , 1 , . . . , τ M , (that is, x 0 τ is bou nded with p robability one), and almost ev ery sequence { σ t } , th ere exists a number α ∈ R such that lim t →∞ x t = α 1 with 1 = [1 , 1 , . . . , 1 ] ⊤ . The mu lti-agent system is said to synchr onize via the algorithm ( 5 ) if fo r any initial essentially b ounded r andom x 0 ∈ R m and almost every sequence { σ t } , lim t →∞ | x i ( t ) − x j ( t ) | = 0 , i, j = 1 , . . . , m . I n particular, if for any initial essentially bo unded random x 0 τ ∈ R m , τ = 0 , 1 , . . . , τ M , and alm ost e very sequence, there exists a P -p eriodic trajectory s ( t ) ( P independen t of the initial v alues and the sequence) such that lim t →∞ | x i ( t ) − s ( t ) | = 0 hold s for all i = 1 , . . . , m , then the multi-agen t system is said to synchr o nize to a P -p eriodic trajectory via the algorithm ( 5 ). In general, consensus can b e regarded as a special case o f syn chronization, where the multi-agen t system synchroniz es at an equilibriu m. As shown in Ref. [ 3 ], in the absence of delays, consensus an d synchro nization are eq ui valent w .r .t. the pro duct of infinite stochastic matrices; that is, whenever a system sy nchronizes, it also reach es consensus. However , we will sh o w in the following sections th at, u nder transmission delays, co nsensus and synchro nization o f the algo rithm ( 5 ) are not equiv alent. Thus, a system can syn chronize without necessarily reaching consensus. Consider the model where the topolog ies are induced by a stochastic process: x t +1 i = m X j =1 G ij ( ξ t ) x t j , i = 1 , . . . , m, (9) where { ξ t } t ∈ N is a stochastic p rocess with a pro bability distribution of the sequ ence P . The results of this pap er ar e b ased on the following lemm a, which is a consequence of Theorem 2 in Ref. [ 3 ]. CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 7 Lemma 2.3. Let η ( · ) den ote the s cramblingness, as defined in ( 7 ). The multi-agent system via the algorithm ( 9 ) r ea c hes consensus if a nd o nly if for P -a lmost every sequence ther e exis t infinitely many disjoint inte ger intervals I i = [ a i , b i ] such that ∞ X i =1 η  b i Y k = a i G ( ξ k )  = ∞ . As a tri v ial extension to a set of SIA matrices, we hav e the next lemma on how to obtain scramblingn ess. Lemma 2.4. [ 34 ] Let Θ ⊂ R m,m be a set of SI A matrices. Ther e exists an inte ger N such that any n -length matrix sequence with n > N picked fr o m Θ : G 1 , G 2 , . . . , G n satisfies η  n Y k =1 G k  > 0 . The following result provides a relation between SIA matrices and spanning trees. Lemma 2 .5. (Lemma 1 in Ref. [ 36 ] ) I f the graph corr espo nding to a stochastic matrix A has a spanning tr ee an d a self-link at one of its r oot vertices, then A is SIA. 3. Main results. W e first consider the multi-agent network without transmission delays: x t +1 i = n X j =1 G ij ( σ t ) x t j , i = 1 , . . . , m. (10) The following theo rem is the main to ol for the p roofs of the main results an d it can be regarded as a realization of Lemm a 2.3 and an extension from Ref. [ 18 ] withou t assuming self-links. Theorem 3.1. F or the system ( 10 ), if ther e exist L ∈ N and δ > 0 such that the δ -g r aph of the matrix pr oduct E  n + L Y k = n +1 G ( σ k ) |F n  (11) has a span ning tr ee an d is a periodic for all n ∈ N almo st surely , then th e multi-agent system r ea c hes a consensus. The proof is gi ven in Sec. 5.1. The main result of [ 18 ] can be r e garded as a consequ ence of T heorem 3.1 , where ea ch no de in the gr aph was a ssumed to h a ve a self-lin k. In th e following, we first study the multi-agen t systems with tran smission delays such that e ach agent is linked to itself w ithout delay and then investigate the g eneral situation wh ere d elays may o ccur also on the self- links. Finally , we give a brief discussion on the consensus algorithm s without self-links. All proof s in this section are placed in Sec. 5. 3.1. Consensus and synchronizat ion with transmission delays. Consider the consensus algorithm ( 6 ), which we rewrite in matrix form as x t +1 = τ M X τ =0 G τ ( σ t ) x t − τ , (12) where G ( σ t ) = [ G τ ij ( σ t )] n i,j =1 . W e assume the following for the matrices G τ ( · ) . A : Each G τ ( σ t ) , τ ∈ τ M , is a measurab le map fro m Ω to the set o f n onnegativ e matrices with respect to F t . 8 W . LU, F . M. A T A Y AND J. JOST Letting y t = [ x t ⊤ , x t − 1 ⊤ , . . . , x t − τ M ⊤ ] ⊤ ∈ R m × ( τ M +1) , we can write ( 12 ) as y t +1 = B ( σ t ) y t , (13) where B ( σ t ) ∈ R ( τ M +1) × m, ( τ M +1) × m has the form B ( σ t ) =        G 0 ( σ t ) G 1 ( σ t ) · · · G τ M − 1 ( σ t ) G τ M I m 0 · · · 0 0 0 I m · · · 0 0 . . . . . . . . . . . . . . . 0 0 · · · I m 0        . Thus, the consensus o f ( 6 ) is equiv alent to that of ( 13 ). As a default labeling, let u s consider the co rresponding graph G ( B ( σ t )) , which has ( τ M + 1) m vertices, which we d enote by { v i,j , i ∈ τ M + 1 , j ∈ m } , where v i,j correspo nds to the ( ( i − 1) m + j )th r o w (or column) of the matrix B ( σ t ) . Theorem 3.2. Assume the conditions A , and suppose ther e exis t µ > 0 , L ∈ N , and δ > 0 such th at G 0 ( σ ) > µI m for all σ ∈ Ω and the δ -g r aph o f E { P n + L k = n +1 G ( σ k ) |F n } has a spanning tr ee for all n ∈ N almo st surely . Then the delayed multi-agent system ( 6 ) r ea c hes consensus. The proof is gi ven in S ec 5.2. In the case that the topological switching is deterministic, a similar result is obtained in the literature [ 24 , 36 ]. Example 3.3. W e give a simp le example to illustrate T heorem 3.2 . Consider a delayed multi-agen t system on a network with 2 vertices an d the maximu m delay is 1 . The system can be written as x t +1 = G 0 ( σ t ) x t + G 1 ( σ t ) x t − 1 , which can further be put into a form without delays y t +1 = B ( σ t ) y t with B ( σ t ) =  G 0 ( σ t ) G 1 ( σ t ) I m 0  . Let us consider the produc t o f tw o matrices B 1 and B 2 : B 1 =     1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0     , B 2 =     1 / 2 0 0 1 / 2 0 1 0 0 1 0 0 0 0 1 0 0     . In the absence of dela ys, the y correspond to G 1 =  1 0 0 1  and G 2 =  1 / 2 1 / 2 0 1  . One can see that th e unio n of the graphs G ( G 1 ) and G ( G 2 ) has span ning trees an d self- connectio ns. The n the proof of Theorem 3.2 says that for some integer L , the product of L successiv e matrices correspon ds to a graph which has a spanning tree and a self-link on the root node. For example, we consider the following matrix product: B 1 B 2 =     1 / 2 0 0 1 / 2 0 1 0 0 1 / 2 0 0 1 / 2 0 1 0 0     . The co rresponding graph has fo ur vertices, which we label as v 1 , 1 , v 1 , 2 , v 2 , 1 , and v 2 , 2 following the scheme defined below Eq. ( 13 ). From Figur e 1 , it can be seen that the grap h CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 9 correspo nding to B 1 B 2 has spanning trees with v 1 , 2 being the root vertex which h as a self-link. So, by Th eorem 3.2 , the system reaches consensu s. 1 , 1 v 1 , 1 v 1 , 1 v 2 , 2 v 2 , 1 v 2 , 1 v 2 , 1 v 1 , 2 v 1 , 2 v 1 , 2 v 2 , 2 v 2 , 2 v 1 B 2 B 2 1 B B F I G U R E 1 . The gr aphs corre sponding to the matrice s B 1 , B 2 , and the matrix prod uct B 1 B 2 , respectively . In some cases delays occ ur at self-links, for example, when it takes time fo r each agent to process its own in formation. Sup pose that the self-linking d elay for each vertex is identical, that is, τ ii = τ 0 > 0 . W e classify each integer t in the discrete-time set N (or the integer set Z ) via mo d ( t + 1 , τ 0 + 1) as the quo tient group of ( Z + 1) / ( τ 0 + 1) . As a d efault set-up, we d enote h i i τ 0 +1 by h i i . Let ˆ G i ( · ) = P j ∈h i i G j ( · ) . For a simplified statement of the result, we provide the following condition B : B .1 Ther e exist an integ er τ 0 > 0 an d a number µ > 0 such that G τ 0 ( σ 1 ) > µI m for all σ 1 ∈ Ω ; B .2 Ther e exis t τ 1 , . . . , τ K excluding the inte gers in h 0 i with gcd ( τ 0 + 1 , τ 1 + 1 , . . . , τ K + 1) = P > 1 such tha t ˆ G j ( σ 1 ) = 0 for all j / ∈ { τ 1 , . . . , τ K } and a ll σ 1 ∈ Ω and the δ -matrix of E { ˆ G τ k ( σ n +1 ) |F n } is no nzer o for a ll n ∈ N a nd k = 1 , . . . , K almost sur ely . Theorem 3. 4. Assume that the condition s A an d B hold , and suppo se ther e e xist L ∈ N and δ > 0 su c h that the δ -graph of E { P n + L k = n +1 ˆ G 0 ( σ k ) |F n } is str ongly conn ected for all n ∈ N almost surely . Then th e system ( 6 ) synchr on izes to a P -p eriodic trajectory . In particular , if P = 1 , then ( 6 ) r eaches consensus. The p roof is given in Sec. 5.3. From this theore m, one can see th at u nder self-linking delays, con sensus is not equivalent to synchro nization. In fact, the delays that o ccur on self-links are essential for the failure to reach consensus. Example 3.5. Theore m 3.4 demands that the δ -graph correspond ing to the matrix E { P n + L t = n +1 ˆ G 0 ( σ t ) |F n } is strongly connected. Th is is st ronger than the condition in The- orem 3.2 , wh ich dem ands th at the co rresponding gra ph has a spannin g tree. W e give an example to show that the stro ng conn ecti v ity is necessary for the reasoning in the proof. Consider a delayed multi-agen t system o n a network with two vertices an d a maximu m 10 W . LU, F . M. A T A Y AND J. JOST 1 , 1 v 1 , 3 v 2 , 1 v 2 , 3 v 1 , 2 v 2 , 2 v 1 , 4 v 2 , 4 v ' 1 G ' 2 G F I G U R E 2 . The graph corr esponding to the matrix product ( 14 ). delay of 3. Consider the fo rm ( 13 ) an d the m atrix B ( · ) . Supp ose that th e state space only contains one state σ 1 as follows: B ( σ 1 ) =             0 0 1 / 3 0 0 1 / 3 0 1 / 3 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0             . Here, τ 0 = 1 . It is clear that the subg raph cor responding to each ˆ G 0 1 , 2 has spannin g trees but is not strongly connected, and that there is a link between the subgraphs corresponding to h 1 i and h 0 i . For the word σ 1 σ 1 · · · σ 1 σ 1 , direct calculations show that the correspo nding matrix prod uct is an analog of the fo llo win g matrix if the len gth of the word is sufficiently long:             1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 0 0 1 0 0 0 0             (14) The correspon ding gr aph is s hown in Figure 2 , using the labeling sch eme fo r the v ertices as defined belo w Eq. ( 13 ). On e can see that it does not have a spanning tree since the vertices v 1 , 2 and v 2 , 2 do not hav e in coming lin ks othe r than self-links. I n fact, the set o f eigenv alues of the matrix B ( σ 1 ) co ntains 1 and − 1 , w hich implies that ( 12 ) with B ( σ t ) can not r each consensus e ven though the condition in Theorem 3.2 is satisfied. 3.2. Consensus and sy nchr onization without self-links. So far th e stab ility re sult is based on the assumption th at e ach agent takes its own state into con siderations when up- dating. In other words, the coup ling matrix has positive diagon als (possibly with delays). There also exist con sensus algo rithms th at are realized by updating each agen t’ s state v ia av eraging its neighbo r’ s states and possibly excluding its own [ 9 ]. In [ 5 ], it is sh o wn tha t consensus can be reached in a static network if each agent can commu nicate with others by CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 11 a dir ected graph and th e coupling grap h is aperiod ic, which can be proved by no nnegati ve matrix theory [ 15 ]. I n the following, we briefly discu ss the g eneral c onsensus alg orithms in networks of stochastically switching topolog ies th at d o no t necessarily have self-lin ks for all vertices. When transmission d elays oc cur , the general algorith m ( 6 ) can b e regarded as increasing dimensions as in ( 13 ). Thus, one can similarly associate ( 13 ) with a new g raph on m × ( τ M + 1) vertices { v ij : i ∈ τ M + 1 , j ∈ m } , d enoted by G ′ ( · ) , where B ( · ) deno tes the link set of G ′ ( · ) , by which v ij correspo nds to the ( i − 1) × ( τ M + 1) + j column an d ro w of B . ˆ B p ( σ 1 ) as the ma trix corresp onding the vertices { v ij : i ∈ h p i , j ∈ m } . Based on theorem 3.1 , we ha ve the following results, which can be proved similarly to Theorems 3.2 and 3.4 . Proposition 3.6. Assume A h olds, and suppose ther e exist L ∈ N and δ > 0 such that the δ -graph of E { Q u + L k = u +1 B ( σ k ) |F u } has a spannin g tr ee and self-link at one r o ot verte x for all n ∈ N almost sur ely . Then the alg orithm ( 10 ) r eaches consensu s. In fact, under the stated conditions, each produ ct E { Q u + L k = u +1 B ( σ k ) |F u } is SIA almost surely; so, this proposition is a direct consequen ce of Theorem 3.1 . In the possible absence of self-links, the following is a consequence of Proposition 3.6 . Proposition 3.7. Assume A and B . 2 ho ld ( B . 1 need not hold). Supp ose ther e e xist L ∈ N and δ > 0 such that the δ -graph of E { Q n + L k = n +1 ˆ B p ( σ k ) |F n } is str on gly connected and has at lea st on e self-link for a ll n ∈ N and p ∈ P almost sur ely , wher e ˆ B p is d efined in the pr oof of Theor em 3.4 , for e xamp le , ( 15 ) in Sec. 5.3. Then the algorithm ( 6 ) synchr on izes to a P -pe riodic trajectory . In particular if P = 1 , then the algorithm ( 6 ) r ea c hes consensus. 4. Ap plications. Adap ted pro cesses are rath er gen eral and inc lude i.i.d p rocesses an d Markov chains as two special cases. Ther efore, the results obtained ab ov e can be d irectly utilized to derive suf ficien t cond itions for the cases where the topology switching a nd d e- lays are i.i.d. o r Marko vian. First, by a stand ard construction as mentioned in Sec. 1, from the prope rty of i.i.d. it follows that E { G ( σ k +1 ) |F k } = E { G ( σ k +1 ) } is a co nstant stochastic matrix. Then, we have the following results. Corollary 4.1. Assume that A ho lds and { σ t } is an i.i.d. pr ocess. Supp ose there exist µ > 0 , L ∈ N , and δ > 0 such that G 0 ( σ ) > µI m for all σ ∈ Ω and the δ -g r a ph o f E { G ( σ 1 ) } has a spanning tr ee. Then the delayed multi- ag ent system via alg orithm ( 6 ) r ea c hes consensus. Corollary 4.2. Assume that A and B h old and { σ t } is an i.i.d . pr ocess. Suppose there exis t L ∈ N and δ > 0 such that the δ -graph of E { ˆ G 0 ( σ 1 ) } is str ong ly con nected for all n ∈ N almost surely . Then th e system ( 6 ) synchr on izes to a P -p eriodic trajectory . In particular , if P = 1 , then ( 6 ) r eaches consensus. Second, we consider the Markovian switching top ologies, n amely , the graph sequence is induced by a homo geneous Markov chain with a stationary distribution and the property of unifor m ergodicity , which is defined as follo ws. Definition 4.3. [ 4 ] A Markov chain { σ t } , defined on { Ω , F } , with a statio nary distribution π and a transition probability T ( x, A ) is called unifo rmly ergodic if X x ∈ Ω k T k ( x, · ) − π ( · ) k → 0 as k → + ∞ , 12 W . LU, F . M. A T A Y AND J. JOST where T k ( · , · ) den otes the k -th itera tion of the transition probability T ( · , · ) , for two pr oba- bility measures µ a nd ν on { Ω , F ) } , and k µ − ν k = sup A∈F | µ ( A ) − ν ( A ) | . From the Markovian property , we have the following results. Corollary 4.4. Assume th at A holds. Let { σ t } be an irr educ ible and aperiodic Markov chain with a un ique invariant measure π . Supp ose { σ t } is uniformly er godic a nd there exist µ > 0 a nd δ > 0 such that G 0 ( σ ) > µI m for all σ ∈ Ω and the δ -graph of E π { G ( σ 1 ) } has a spanning tr ee. The n the delayed multi-ag ent system( 6 ) r eaches consensus. Pr oof. From the Markovian property , we ha ve E { 1 L n + L X t = n +1 G ( σ t ) |F n } = E { 1 L n + L X t = n +1 G ( σ t ) | σ n } . If { σ t } is uniformly ergodic, then lim L → + ∞ E { 1 L n + L X 1= n +1 G ( σ t ) | σ n } = lim L → + ∞ 1 L L X i =1 Z Ω G ( y ) T i ( σ n , dy ) = Z Ω G ( y ) π ( dy ) = E π [ G ( σ 1 )] . Since the convergence is uniform , ther e exits L su ch that the δ / 2 -g raph correspo nding to E { (1 /L ) P n + L t = n +1 G ( σ t ) |F n } h as a spa nning tree almost surely . From Theorem 3.2 , the conclusion can be deriv ed. Corollary 4. 5. Assume that A a nd B ho ld, an d let { σ t } be an irr educib le and aperiodic Markov chain with a uniqu e in va riant measur e π . Supp ose that { σ t } is uniformly ergodic and ther e exists δ > 0 such that the δ -graph of E π { ˆ G 0 ( σ 1 ) } is str ongly connected. Then the system ( 6 ) synchr onizes to a P - periodic trajectory . In pa rticular , if P = 1 , then ( 6 ) r ea c hes consensus. These c orollaries can be p roved directly from Th eorems 3.4 in the same way as Corol- lary 4.4 . It can be seen that the a ho mogeneous Markov c hain with finite state space an d unique in variant distribution is uniformly ergodic. He nce, the results of Corollaries 4.4 and 4.5 hold for this scenario. 5. Proofs o f the main results. In the following, the couplin g matrix B ( · ) in the delayed system ( 13 ) is written in the following block form: B ( σ t ) =      B 1 , 1 ( σ t ) B 1 , 2 ( σ t ) · · · B 1 ,τ M +1 ( σ t ) B 2 , 1 ( σ t ) B 2 , 2 ( σ t ) · · · B 2 ,τ M +1 ( σ t ) . . . . . . . . . . . . B τ M +1 , 1 ( σ t ) B τ M +1 , 2 ( σ t ) · · · B τ M +1 ,τ M +1 ( σ t )      ∈ R ( τ M +1) m, ( τ M +1) m with B ij ( σ t ) ∈ R m,m , i, j ∈ τ M + 1 . For two ind ex sets I and J , we denote by [ B ( σ t )] I ,J the sub-m atrix of B ( σ t ) with row index set I and column ind e x set J . For an n -len gth word σ = ( σ k ) n k =1 in the stochastic pro cess, we use B ( σ ) to represent the matrix pro duct Q n i =1 B ( σ i ) . One ca n see that th e structure of th e matrix B ( σ t ) has the following proper- ties: (1). Each B i,i − 1 = I m for all i ≥ 2 ; (2). B i,j = 0 for all i ≥ 2 and j 6 = i − 1 . The se proper ties ar e essential for the following proofs. As the same way d efined below Eq. ( 13 ), let u s consider th e corr esponding graph G ( B ( σ t )) , which has ( τ M + 1 ) m vertices, which we denote by { v i,j , i ∈ τ M + 1 , j ∈ m } , where v i,j correspo nds to the ( i − 1) m + j r o w of the matrix B ( σ ) . CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 13 W e denote the following finitely generated period ic group: h i 1 , i 2 , . . . , i K i j := { p : p = K X l = k i k p k mo d j, p k ∈ Z } . If the se numb ers are be picked in a finite integer set, fo r instance, { 1 , . . . , τ M + 1 } in the present paper, then h i 1 , i 2 , . . . , i K i j denotes the set h i 1 , i 2 , . . . , i K i j T τ M + 1 unless specified o therwise. As a default setup, h i i d enotes h i i τ 0 +1 where τ 0 is th e self-link ing delay as in ( 12 ). W e will sometimes be interested in whether an element in a matrix i s zero or not, regardless of its actual v alue. 5.1. Proof of Theorem 3.1 . From the co ndition in this theorem, we can see that th e δ - matrix of E { Q n + L k = n +1 G ( σ k ) |F n } is SIA f or all n ∈ N . Le mma 2.4 states that th ere exists N ∈ N such that the product of any N SIA matrices in R m,m is scramb ling. No te that E  n + N L Y t = n +1 G ( σ t ) |F n  = E  · · · E  E  n + N L Y t L = n +( N − 1) L +1 G ( σ t L ) |F n +( N − 1) L  n +( N − 1) L Y t L − 1 = n +( N − 2) L +1 G ( σ t L − 1 ) |F n +( N − 2) L  · · · n + L Y t 1 = n +1 G ( σ t 1 ) |F n  , since {F t } is a filtratio n. This implies that the re exists a p ositi ve constant δ 1 < δ N such that the δ 1 -graph of E { Q n + N L t = n +1 G ( σ t ) |F n } is scrambling. So, fr om Lemma 3.12 in Ref. [ 18 ], there exist δ ′ > 0 an d M 1 ∈ N such th at P  η  n + M 1 N L Y t = n +1 G ( σ t )  > δ ′ |F n  > δ ′ , ∀ n ∈ N . Let C k = Q ( k +1) M 1 N L t = kM 1 N L +1 G ( σ t ) . W e can c onclude that for almost every sequence of { σ t } , it holds that lim K →∞ K X k =1 P  η ( C k ) > δ ′ |F kN L  > lim K →∞ K × δ ′ = + ∞ . From Lem ma 2.1 , we can conclu de that the events { η ( C k ) > δ ′ } , k = 1 , 2 , . . . , occu r infinitely of ten almost surely . Th erefore, we can complete the p roof directly fr om Lemma 2.3 . 5.2. Proof of Theorem 3.2 . The proo f of this the orem is based on the structural character- istics of the product of matrices B ( · ) . W e deno te by [ B ( · )] ij the R m,m sub-matrix of B ( σ ) in the position ( i, j ) . W e first sho w by th e f ollo wing lem ma that the graph correspond ing to the prod uct of more th an τ M + 1 successiv e matrices B ( σ t ) , as defined by ( 13 ), has a span- ning tree and self- link at one roo t vertex. Thus, we can prove Theor em 3.2 by em ploying Theorem 3.1 . Lemma 5.1 . Under the con ditions in Theor em 3.2 , for a ny n -len gth word σ = ( σ i ) n i =1 with n ≥ τ M + 1 , ther e exis ts µ 1 > 0 such that (i). [ B ( σ )] i, 1 ≥ µ n 1 I m ; (ii). P τ M +1 j =1 [ B ( σ )] 1 ,j ≥ µ n 1 P τ M +1 j =1 P n k =1 G j ( σ k ) . 14 W . LU, F . M. A T A Y AND J. JOST Pr oof. W e choo se 0 < µ 1 < µ , where µ is defined in Theo rem 3.2 . (i). For a word σ = ( σ i ) n i =1 with n ≥ τ M + 1 , [ B ( σ )] i, 1 = X i 1 ,...,i n [ B ( σ n )] i,i 1 [ B ( σ n − 1 )] i 1 ,i 2 · · · [ B ( σ 1 )] i n, 1 ≥  n Y k = n − i +2 [ B ( σ k )] k + i − n,k + i − n − 1  n − i +1 Y k =1 [ B ( σ k )] 1 , 1  = n − i +1 Y k =1 [ B ( σ k )] 1 , 1 ≥ µ n 1 I m since [ B (  )] k + i − n,k + i − n − 1 = I m for all k ≥ n − i + 2 an d [ B (  )] 1 , 1 ≥ µI m ≥ µ 1 I m for all  ∈ Ω . (ii). L et j ∈ τ M + 1 and t 0 ∈ n . If t 0 ≥ j , we have X l [ B ( σ )] 1 ,l = X i 1 ,...,i n ,l [ B ( σ n )] 1 ,i 1 [ B ( σ n − 1 )] i 1 ,i 2 · · · [ B ( σ 1 )] i n ,l ≥  n Y k = t 0 +1 [ B ( σ k )] 1 , 1  [ B ( σ t 0 )] 1 ,j  t 0 − 1 Y l = t 0 − j +2 [ B ( σ l )] l − t 0 + j,l − t 0 + j − 1   t 0 − j +1 Y p =1 [ B ( σ p )] 1 , 1  ≥ µ n 1 [ B ( σ t 0 )] 1 ,j , since [ B (  )] 1 , 1 ≥ µ 1 I m , [ B (  )] l − t 0 + j,l − t 0 + j − 1 = I m for all l ≥ t 0 − j + 2 for all  ∈ Ω ; whereas if j > t 0 , we similarly have X l [ B ( σ )] 1 ,l ≥  n Y k = t 0 +1 [ B ( σ n )] 1 , 1  [ B ( σ t 0 )] 1 ,j  t 0 − 1 Y l =1 [ B ( σ l )] l + j − t 0 +1 ,l + j − t 0  ≥ µ n 1 [ B ( σ t 0 )] 1 ,j . Summing the right-hand side of the a bove inequ ality with respect to t 0 and j proves (ii). Pr oof of Theorem 3.2 . Let u s consider the µ n 1 -graph of B ( σ ) for all σ = ( σ t ) n t =1 with n ≥ τ M + 1 , as defin ed in Lemm a 5.1 . The item (i) in Lemma 5.1 indicates that for each vertex v i,j with i ≥ 2 a nd j ∈ m , there exist a path f rom vertex v 1 ,j to v i,j : ( v 1 ,j , v 2 ,j , . . . , v i,j ) . From item (ii) in Lem ma 5.1 and the conditions in Theor em 3.2 , one can see that th ere exits δ > 0 and L ∈ N such that th e δ -g raph of P l [ E { Q n + L t = n +1 B ( σ t ) |F n } ] 1 ,l has span- ning trees a nd self-links. Let G be th e r andom variable correspon ding to th e δ -g raph of E { Q n + L t = n +1 B ( σ t ) |F n } an d G ′ be the rand om variable correspon ding to the δ -graph of P l [ E { Q n + L t = n +1 B ( σ t ) |F n } ] 1 ,l . Then, for alm ost every graph G ′ , there exists an in dex j 0 ∈ m such that for any j , th ere exists a path ( j 0 , j 1 , . . . , j K − 1 , j ) to access j . T his im- plies that for almo st every graph G , there exists a path from v 1 ,j 0 to v 1 ,j . Thus, v 1 ,j 0 can access all vertices v i,j , i = 1 , . . . , τ M + 1 , since v 1 ,j can access all v i,j for τ M + 1 ≥ i ≥ 2 by a directe d link and v 1 ,j 0 and h as self-link, noting that G 0 ( · ) has positiv e diago nals. Therefo re, for alm ost ev ery grap h G , it has a spann ing tree and the vertex v 1 ,j 0 is one of the roots. From Lemma 2.5 , one can see that E { Q n + L t = n +1 B ( σ t ) |F n } is SIA almost surely . According to Theorem 3.1 , the system ( 10 ) reaches consensus. This p roves the th eorem. CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 15 5.3. Proof of Theorem 3.4 . Outline o f t he pr o of: For a better un derstanding of the pro of, we first give the following sketch. W e start th e pro of by definin g a permu tation matrix Q ∈ R τ M +1 ,τ M +1 correspo nding to the perm utation sequ ence from (1 , 2 . . . , τ M + 1) to ( h 1 i , h 2 i , . . . , h P i ) . T hen we sho w by the lemma that follows that the matrix B ( σ t ) can be transform ed into the following form: [ Q ⊗ I m ] B ( σ t )[ Q ⊗ I m ] ⊤ =      ˆ B 1 ( σ t ) 0 · · · 0 0 ˆ B 2 ( σ t ) · · · 0 . . . . . . . . . . . . 0 0 · · · ˆ B P ( σ t )      , (15) where ⊗ stand s for the K ronecker produ ct and ˆ B p ( σ t ) = B hh p i| P i , hh p i| P i ( σ t ) . By the permutatio n Q , we can re write the cou pled system ( 5 ) as ˆ y t +1 = ˆ B ( σ t ) ˆ y t , (16) where ˆ y t = [ Q ⊗ I m ] y t and ˆ B ( · ) = [ Q ⊗ I m ] B ( · )[ Q ⊗ I m ] ⊤ . Th is system can be di vid ed into P subsystem as ˆ y t +1 p = ˆ B p ( σ t ) ˆ y t p , p ∈ P , (17) where ˆ y t p correspo nds to [ y t ] hh p i| P i . So, it is sufficient to p rove the following claim to complete this proof from Lemma 3.1 : Claim 1: For ea ch p ∈ P , there exists δ ′ > 0 and L ∈ N such that the δ ′ -graph of the matrix E  n + L Y t = n +1 ˆ B p ( σ t ) |F n  (18) has a spanning tree for all n ∈ N almost surely . The p roof of this theorem is also ba sed o n the structural characteristics of the p roduct of matrice s B ( · ) . By the lemmas b elo w , we are to show the permutatio n form ( 15 ) can be guaran teed. Lemma 5 .2. Und er the con ditions of Theor em 3.4 , for any ( τ 0 + 1 ) - length wor d σ = ( σ k ) τ 0 +1 k =1 , ther e exists some µ 1 > 0 su c h that the following hold: (i). [ B ( σ )] i,i ≥ µ τ 0 +1 1 I m for all i ∈ τ 0 + 1 ; (ii). [ B ( σ )] j,j − ( τ 0 +1) ≥ I m for all j ≥ τ 0 + 2 ; (iii). P l ∈h 1 − j i [ B ( σ )] τ 0 +2 − j,l ≥ µ τ 0 +1 1 ˆ G 0 ( σ j ) for all j ∈ τ 0 + 1 ; (iv). P l ∈h i +( τ +1) i [ B ( σ )] i,l ≥ µ τ 0 +1 1 [ B ( σ τ 0 +2 − i )] 1 ,τ +1 for all i ∈ τ 0 + 1 and τ ∈ τ M . Pr oof. W e c hoose 0 < µ 1 < µ . (i). For any i ∈ τ 0 + 1 , we hav e [ B ( σ )] i,i = X i 1 ,...,i τ 0 [ B ( σ τ 0 +1 )] i,i 1 [ B ( σ τ 0 )] i 1 ,i 2 · · · [ B ( σ 1 )] i τ 0 ,i ≥  τ 0 +1 Y p = τ 0 +3 − i [ B ( σ p )] p + i − 1 − τ 0 ,p + i − 2 − τ 0  [ B ( σ τ 0 − i +2 )] 1 ,τ 0 +1  τ 0 − i +1 Y q =1 [ B ( σ q )] q + i,q + i − 1  ≥ µI m ≥ µ τ 0 +1 1 I m since [ B (  )] i +1 ,i = I m and [ B (  )] 1 ,τ 0 +1 ≥ µI m for all  ∈ Ω and i ∈ τ M . 16 W . LU, F . M. A T A Y AND J. JOST (ii). For any j ≥ τ 0 + 2 , we have [ B ( σ )] j,j − ( τ 0 +1) = X i 1 ,...,τ 0 [ B ( σ τ 0 +1 )] j,i 1 [ B ( σ ( τ 0 ))] i 1 ,i 2 · · · [ B ( σ 1 )] i τ 0 ,j − ( τ 0 +1) ≥ τ 0 +1 Y k =1 [ B ( σ k )] k + j − τ 0 − 1 ,k + j − τ 0 − 2 = I m since [ B (  )] i +1 ,i = I m for all i ≥ 2 and  ∈ Ω . (iii). For any i ∈ τ 0 + 1 , we have X i 1 ,...,i τ 0 ,k [ B ( σ )] i,i +( τ 0 +1) k = X k [ B ( σ τ 0 +1 )] i,i 1 [ B ( σ τ 0 )] i 1 ,i 2 · · · [ B ( σ 1 )] i τ 0 , ( τ 0 +1) k + i ≥  i Y k =2 [ B ( σ τ 0 − i + k +1 )] k,k − 1  [ B ( σ τ 0 − i +2 )] 1 , ( k +1)( τ 0 +1)  ( k +1)( τ 0 +1) Y l = i +( τ 0 +1)( k +1) − τ 0 [ B ( σ l − k ( τ 0 +1) − i )] l,l − 1  ≥ [ B ( σ τ 0 +2 − i )] 1 , ( k +1)( τ 0 +1) for all k ≥ 0 . Sum ming the right-hand side with respect to k and letting j = τ 0 + 2 − i , we have P l ∈h 1 − j i [ B ( σ )] τ 0 +2 − j,l ≥ P l ∈h τ 0 +1 i [ B ( σ j )] 1 ,l . (iv). Let j = τ 0 + 2 − i . If j ≥ τ , X k [ B ( σ )] τ 0 +2 − j,τ 0 +2 − j +( τ +1)+( τ 0 +1) k ≥  τ 0 +1 Y p = j +1 [ B ( σ p )] p − j +1 ,p − j  [ B ( σ j )] 1 ,τ +1  j − 1 Y q = j − τ [ B ( σ q )] q + τ +2 − j,q + τ +1 − j  [ B ( σ j − τ − 1 )] 1 ,τ 0 +1  j − τ − 2 Y l =1 [ B ( σ l )] l + τ + τ 0 +3 − j,l + τ + τ 0 +2 − j  ≥ µ [ B ( σ j )] 1 ,τ +1 ; whereas if j < τ , X k [ B ( σ )] τ 0 +2 − j,τ 0 +2 − j +( τ +1)+( τ 0 +1) k ≥  τ 0 +1 Y p = j +1 [ B ( σ p )] p − j +1 ,p − j  [ B ( σ j )] 1 ,τ +1  j − 1 Y q =1 [ B ( σ q )] q + τ +2 − j,q + τ +1 − j  ≥ [ B ( σ j )] 1 ,τ +1 . These calculations complete the proof of the lemma. Lemma 5.3. Un der the cond itions of Th eor em 3.4 , consider a n L ( τ 0 + 1) -leng th wor d ˜ σ = ( ˜ σ 1 , . . . , ˜ σ L ) , where each ˜ σ l = ( σ l,i ) τ 0 +1 i =1 is a ( τ 0 + 1) -len gth wor d. If L ≥ τ M + 1 , then ther e exists µ 1 > 0 such that (i). [ B ( ˜ σ )] j,i ≥ µ ( τ 0 +1) L 1 I m for all j ∈ h i i and i ∈ τ 0 + 1 ; (ii). P l ∈h i i [ B ( ˜ σ )] τ 0 +2 − j,l ≥ µ ( τ 0 +1) L 1 P k ˆ G 0 ( σ k,j ) for all j ∈ τ 0 + 1 ; (iii). P j ∈h i + τ +1 i [ B ( ˜ σ )] i,j ≥ µ τ 0 +1 1 P l ∈h τ +1 i [ B ( ˜ σ τ 0 +2 − i )] 1 ,l for all i ∈ τ 0 + 1 and τ ∈ τ M ; (iv). If τ ′ is such that τ ′ + 1 / ∈ h τ 0 + 1 , τ 1 + 1 , . . . , τ K + 1 i and [ B ( σ 1 )] 1 , h τ ′ +1 i = 0 for all σ 1 ∈ Ω , then [ B ( ˜ σ )] i, h i + τ ′ +1 i = 0 fo r all i ≥ 1 . CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 17 Pr oof. W e p ick some µ 1 < µ . (i). For j ≤ τ 0 + 1 , the proof is similar to the pr oof of item (i) of Lemma 5.2 . For j ≥ τ 0 + 2 , we hav e [ B ( ˜ σ )] j,i ≥  L Y l = l 1 [ B ( ˜ σ l )] j − ( L − l )( τ 0 +1) ,j − ( L − l +1)( τ 0 +1)  l 1 Y p =1 [ B ( ˜ σ p )] i,i  ≥ µ ( τ 0 +1) L 1 I m , where l 1 = L + 1 − ( j − i ) / ( τ 0 + 1) is an integer (noting j ∈ h i i ), since [ B ( ˜ σ l )] j − ( L − l )( τ 0 +1) ,j − ( L − l +1)( τ 0 +1) ≥ I m holds here, as mentioned in Lemma 5.2 (ii). The items ( ii) and (iii) can be proved by similar arguments as in the pro of o f items (iii) and (i v) of Lemma 5.2 . It rema ins to prove item (i v) . I n the follo wing, we will prove a slightly mor e gen eral r esult, namely th at [ B ( σ )] i, h i + τ ′ +1 i = 0 for all words σ h a vin g len gth L ∈ h τ 0 + 1 i . Let σ = ( σ i ) L i =1 be an arbitrary L -length w ord. W e calcu late [ B ( σ )] i,j with j ∈ h i + τ ′ + 1 i as a sum of se veral matrix product terms: [ B ( σ )] i,j = X i 1 ,...,i L − 1 [ B ( σ L )] i,i 1 [ B ( σ L − 1 )] i 1 ,i 2 · · · [ B ( σ 1 )] i L − 1 ,j . Since any zero factor y ields zero prod uct, we av oid zero factors in the calc ulations. That is, in the expression above, only facto rs of th e form [ B ( σ l )] i +1 ,i and [ B ( σ l )] 1 ,j can o ccur where j ∈ h i + τ ′ + 1 i an d τ ′ + 1 / ∈ h τ 0 + 1 , τ 1 + 1 , . . . , τ K + 1 i . Thus, letting j 1 = i , we have [ B ( σ )] i,j = X j 1 ,...,j V ,V  V Y l =1  L − P l − 1 p =1 j p Y k l = L − P l p =1 j p +2 [ B ( σ k l )] P l p =1 j p + k l − L, P l p =1 j p + k l − L − 1  [ B ( σ L − P l p =1 j p +1 )] 1 ,j p +1  L − P V p =1 j p Y k V +1 =1 [ B ( σ k V +1 )] L − P V p =1 j p + k V +1 , P V p =1 L − P V p =1 j p + k V +1 − 1  , where each j p ∈ h τ 0 + 1 , τ 1 + 1 , . . . , τ K + 1 i . Suppose that the matrix product is nonzero. Then j = P V p =1 j p − L , i.e., h ( i + τ ′ + 1) − ( P V p =1 j p − L ) i = 0 , which implies h τ ′ + 1 − P V p =2 j p + L i = 0 . This means that τ ′ + 1 ∈ h τ 0 + 1 , τ k + 1 : k = 1 , . . . , K i , which con tradicts the conditio n τ ′ + 1 / ∈ h τ 0 + 1 , τ k + 1 : k = 1 , . . . , K i . The lemm a is proved. Pr oof of Theor em 3.4 . Consider the graph ˆ G δ ( σ t ) = { ˆ V , ˆ E ( σ t ) } o n ( τ M + 1 ) m vertices correspo nding to the δ -grap h of the m atrix B ( σ t ) a s defined at the beginning of this section . For L ∈ N as fixed in the main condition of Theorem 3.4 and an arbitrary fixed m ∈ N , let B = E { Q n + L t = n +1 B ( σ t ) |F n } and ˆ G δ be the random variable picked in the δ -graphs of B . First, we divide the graph ˆ G δ into τ M + 1 subgr aphs: G δ k = {V k , E k ( σ t ) } , k ∈ τ M + 1 , where V k = { v k,i : i ∈ m } correspon ds to the rows or column s of B k,k and the vertex v k,i correspo nds the i -th row o r column of the matrix B k,k . Then, in te grate the subgraph s {G δ k } τ M +1 k =1 into τ 0 + 1 subg raphs: G ′ δ l = { V ′ l , E ′ l } , l ∈ τ 0 + 1 , wher e V ′ l = S k ∈h l i V k , l ∈ τ 0 + 1 and E ′ l correspo nds to the i ntra-links in V ′ l . Let E l 1 ,l 2 denote the inter-links from the subgr aph of V ′ l 2 to the subg raph V ′ l 1 . Lemma 5.3 (i) implies that for ea ch l ∈ τ 0 + 1 , there mu st exist a lin k from v l,i to v k,i in the subgrap h G ′ δ l ( · ) , for each vertex v k,i ∈ V k with k > l an d k ∈ h l i . Similarly to the the p roof of Theo rem 3.2 , the main condition of Theo rem 3.4 and items (ii) and (iii) in Lemma 5 .3 imp ly that there exist δ 1 > 0 and L ∈ N such that the su bgraph G ′ δ 1 l is stro ngly connected, con sequently ha ving a spanning tree, and each vertex in V l is one o f the roo ts in G ′ δ 1 l and has a self-link almost surely for all l ∈ τ 0 + 1 . 18 W . LU, F . M. A T A Y AND J. JOST Second, accordin g to gcd ( τ 0 + 1 , τ k + 1 : k ∈ K ) = P , we integrate the subgr aphs G ′ δ 1 l for all l ∈ τ 0 + 1 , into P subg raphs, denoted by ˜ G δ 1 p = { ˜ V p , ˜ E p } , p ∈ P by ˜ V p = {V ′ j : E j,p 6 = ∅} . The items (ii) and (iii) in Lemma 5.3 and the secon d item in condition B indicate that the δ 1 -matrix of P j ∈h τ k +1: k =0 , 1 ,...,K i B l,l + j is positive for all l ∈ τ 0 + 1 . This imp lies that ther e exists at least on e link f rom G ′ δ 1 l + j to G ′ δ 1 l and this link end in V l . So, in the gr aph G ′ δ 1 , the root vertex in G ′ δ 1 l + j can reach all vertices in G ′ δ 1 l since each vertex in V l is a root vertex in G ′ δ 1 l . Th is leads to the conclusion th at V ′ j ⊂ ˜ V l provided j − l ∈ h τ k + 1 : k = 0 , 1 , . . . , K i . Also, we can conclude that each root vertex in G ′ l + j δ 1 can reach all vertices in G ′ δ 1 l , by item (i) in Lemma 5.3 . There fore, we can co nclude tha t ˜ V p = S l ∈h p i P V ′ p and each ˜ G p has a spannin g tree almost su rely . This proves Claim 1. Moreover , there exists a vertex with self-link in V i , i ∈ τ 0 + 1 and i ∈ h p i P , as one of its roots, in ˆ G δ 1 . So, accor ding to the arbitrar iness of integer n , we can co nclude that the δ 1 -graph of E { Q n + L t = n +1 ˆ B p ( σ t ) |F n } is SIA almost surely for all n ∈ N . Finally , accor ding to the second item in cond ition B and th e (iv) item in Lemma 5.3 , one can conclud e that there a re no link s between the graph ˜ G δ p for different p ∈ P for any δ ≥ 0 . So, b y a p ermutation matrix Q cor responding to the permutatio n seq uence fro m (1 , 2 . . . , τ M + 1) to ( h 1 i , h 2 i , . . . , h P i ) , [ Q ⊗ I m ] B ( σ t )[ Q ⊗ I m ] ⊤ has the form ( 15 ). By Theor em 3.1 , we can conclude that ( 17 ) reaches consensus for all p = 1 , . . . , P , but conv erges to different values except fo r initial values in a set of Leb esgue measur e zero. Therefo re, x t can synchro nize and con verge to a P -perio dic trajectory . This co mpletes the proof of Theorem 3.4 . 6. Conclusions. In this pape r we have studied t he con vergence of the consensus algorithm in mu lti-agent systems with stochastically switching to pologies and tim e delays. W e h a ve shown that consen sus can be obta ined if the graph cor responding to the co nditional ex- pectations of the coupling m atrix produc t in consecutive times has spannin g tr ees almost surely and self-lin ks a re p ossible. With m ultiple delays, if self-links always exist and are instantaneou s (und elayed), then consensus can be guaranteed for arbitrary bounded delays. Moreover , when the self-lin ks are also delayed, we have shown th e phenom enon that the algorithm may not reach consensus but instead may synchronize to a periodic trajectory ac - cording to the delay patterns. Finally , we h a ve briefly studied consensus algorithms witho ut self-links. W e hav e presented se vera l results for i.i.d. and Markovian switching topolog ies as special cases. Acknowledgments. W . Lu is supported by the National Ke y Basic Research and De velop- ment Program (No. 201 0CB731403), the Na tional Natu ral Science s Foundation of China under Gran t No. 6080 4044, the Foundation for the Author of National Ex cellent Doctoral Dissertation of China No. 2 00921, SGST 09DZ2 272900, and LMNS, Fudan Univ ersity . REFERENCES [1] P .-A. Bliman and G. Ferrari-Tre cate, A ver ag e consensus prob lems in networks of agents with delayed com- municati ons , Automatica , 44 (2008), 1985–1995. [2] M. Cao, A. S. Morse and B . D. O. Anderson, R eac hing a consensus in a dynamicall y changin g en vir onment: a graphic al appro ach , SIAM J. Control Optim., 47 (2008), 575–600. [3] S. Chatterje e and E. Seneta, T owar ds consensus: Some con ver gence theor ems on re peated avera ging , J. Appl. Prob ., 14 (1977), 89–97. [4] O. Chilina, “f-uniform ergodici ty of Markov chains”, Supervised Project, Un versity of T oronto, 2006. [5] M. H. DeGroot, Reachin g a consensus , J. Amer . Statist. Assoc., 69 (1974), 118–121. CONSENSUS AND SYNCHRONIZA TION IN NET WORKS 19 [6] D. V . Dimarogonasa and K. H. Johansson, Stability analysi s for multi-ag ent systems using the inciden ce matrix: Quantized communic ation and formation contr ol , Automatica , 46 (2010), 695–700. [7] R. Durrett, “Probabilit y: Theory and Examples”, 3 r d editi on, Belmont, CA: Duxbury Press, 2005. [8] F . Fagnani and S. Z ampi eri, A vera ge consensus with pack et dro p communicatio n , SIAM J. Control Optim., 48 (2009), 102–133. [9] L. Fang, P . J. Antsaklis and A. Tzimas, Asynchr onous consensus prot ocols: Prelimin ary resul ts, simula- tions and open questi ons , Procee dings of the 44th IEEE Conf. Decision and Control, the Europ. Control Conferen ce (2005), 2194–2199. [10] J. A. Fax and R. M. Murray , Informati on flow and cooperative contr ol of vehic le formations , IEE E Trans. Autom. Control, 49 (2004), 1465–1476. [11] C. Godsil and G. Royle , “ Algebraic graph theory” , Springer-V erlag, New Y ork, 2001. [12] J. H aj nal, The ergo dic pro perties of non-homoge neous finite Markov chains , Proc. Camb . Phil. Soc., 52 (1956), 67–77. [13] J. Ha jnal, W eak erg odicit y in non-homog eneous Marko v chains , Proc. Camb . Phil. Soc., 54 (19 58), 233–24 6. [14] Y . Hatano and M. Mesbahi, Agr eement over random networks , IEEE Trans. Autom. Control, 50 (2005), 1867–1872. [15] R. A. Horn and C. R. Johnson, “Matrix analysis”, Cambridge Univ ersity Press, 1985. [16] Y . Kuramoto, “Chemical Oscillat ions, wav es, and turbul ence” , Springer-V erlag, New Y ork, 1984. [17] J. Lin, A. S. Morse and B. D. O. Anderson, The multi-a gent rendezvou s pr oblem P art 2: The asynchr onous case , SIAM J. Control Optim., 46 (2007), 2120–2147. [18] B. Liu, W . Lu and T . Chen, Consensus in networks of multiag ents with switch ing topologi es modeled as adapted stochast ic proc esses , SIAM J. Control Optim., 49 (2011), 227–253. [19] W . Lu, F . M. Atay a nd J. Jost, Sync hroni zation of disc r ete-ti me networks with time-varying couplin gs , SIAM J. Math. Analys., 39 (2007), 1231–1259. [20] W . Lu, F . M. Atay a nd J. Jost, Chaos s yn chr onization in networks of coupled maps with time-varyi ng topolo- gies , Eur . Phys. J. B, 63 (2008), 399–406. [21] N. A. L ynch, “Distr ibut ed algorithms”, CA: Morgan Kaufma nn, San Francisco, 1996. [22] W . Ni and D. Z. Cheng, Lead er-fo llowin g consensus of multi-ag ent systems under fixed a nd switc hing topolo- gies , Systems & Control Letters, 59 (2010), 209–217. [23] W . Michie ls, C.-I. Mor ˘ arescu and S.-I. Niculescu, Consensus prob lems with distrib uted delays, with appli- cation to traf fic flow models , SIAM J. Control Optim., 48 (2009), 77–101. [24] L. Moreau, Stabil ity of continuou s-time distribut ed consensus algorithms , 43rd IEE E Conference on Deci- sion and Control, 4 (2004), 3998–4003. [25] L. Moreau, Stabilit y of multiag ent systems with time-de pendent communic ation links , IE EE Trans. Autom. Control, 50 (2005), 169–182. [26] R. Olfati -Saber and J. S. Shamma, Consensus filter s for sensor networks and distrib uted sensor fusion , 44th IEEE Conferenc e on Decision and Control 2005, and 2005 European Control Conferenc e CDC-ECC ’05. 6698–6703. [27] R. Olfati -Saber , J. A. Fax an d R. M. Murray , Consensus and cooperat ion in network ed multi-a gen t systems , Proceedi ngs of the IEEE, 95 (2007), 215–233. [28] R. Olfati-Sa ber and R. M. Murray , Consensus pr oblems in networks of agents with switching topology and time-de lays , IEEE Trans. Aut om. Control, 49 (2004),1520–1533 . [29] A. Pikovsk y , M. Rosenblum and J. Kurt hs, Sync hroni zation: A universal concept in nonlinear scie nces , Cambridge Uni ver sity Press, 2001. [30] J. Shen, A geome tric approa ch to ergo dic non-homog eneou s Markov cha ins , W av elet Anal. Multi. Meth., LNP AM, 212 (2000), 341–366. [31] A. T ahbaz -Saleh i and A. Jadbabaie , A necessary and suffi cient condition for consensus over rando m net- works , IEEE Trans. Aut om. Control, 53 (2008), 791–795. [32] T . V icsek, A. Czir ´ ok, E . Ben-Ja cob, I. Cohen and O. Shochet, Novel type of phase transitio n in a system of self-driv en particles , Phys. Rev . Lett, 75 (1995), 1226–1229. [33] A. T . Win free, “The Geometry of biologi cal time”, S pri nger V erlag, Ne w Y ork, 1980. [34] J. W olfowit z, Produc ts of indecomposa ble , aperiodic , stochasti c matrices , Proceedings of AMS, 14 (1963), 733–737. [35] C. W . Wu, Sync hr onization and conv er ge nce of linear dynamics in random direc ted networks , IE EE Tra ns. Autom. Control, 51 (2006), 1207–1210. [36] F . Xiao and L. W ang, Consensus pr otoco ls for discre te-ti me multi-ag ent systems with time-varying delays , Automatic a, 44 (2008), 2577–2582. [37] F . Xiao, L . W ang, Asynchr onous consensus in continuous-time multi-ag ent systems with switchin g topology and time-varyi ng delays , IEEE Transac tions on Automatic Control, 53 (2008), 1804–1816. 20 W . LU, F . M. A T A Y AND J. JOST [38] Y . Zhang and Y .-P . T ian, Consent abili ty and pr otocol de sign of multi -ag ent systems w ith sto cha stic switc hing topolo gy , Automatica, 45 (2009), 1195–1201. E-mail addre ss : wenlian@fudan.edu .cn E-mail addre ss : atay@mis.mpg.de E-mail addre ss : jost@mis.mpg.de