The compositional construction of Markov processes

The comp ositional construction of Mark o v pro cesses L. de F rancesco Albasini N. Sabadini R.F.C. W alters No v em b er 19, 2018 Abstract W e describe an algebra for comp osi ng automata in whic h th e actions hav e probabilities. W e illustrate by sho wing how to calculate the prob a- bilit y of reaching deadlo c k in k steps in a model of the cla ssical Dining Philosopher problem, and show, using the P erron-F rob enius Theorem, that this probabilit y tends to 1 as k tends to infinity . 1 In tro duction The idea of this pap er is to in tro duce in to the alg ebra of automata introduced in [4] pr obabilities on the actions. This p ermits a compo sitional description of probabilistic pro cesses, in par ticular of Markov chains, and for this rea son we call the automata we in tro duce Markov automata . W e define a Marko v automaton with a given set A of “ signals on the left int erfac e”, and set B of “signa ls on the rig h t interface” to consist of a Ma rk ov matrix Q (whose rows and columns are the states ) which is the sum Q = Q a 1 ,b 1 + Q a 2 ,b 1 + · · · + Q a m ,b n of non-neg ativ e matr ices Q a i ,b j whose elements ar e the pro babilities of trans i- tions b et ween states for whic h the signals a i and b j o ccur o n t he interfaces. In a ddition, each alphab et is r equired to contain a sp ecial symbol ε ( the null signal) and the matrix Q ε A ,ε B is required to hav e row sums strictly p ositive. There exist alr eady in the literatur e mo dels o f proba bilistic pro cesses, for example the a utomata o f Rabin [6], which a re howev er no n-compositiona l. An- other mo del whic h do es includes compositio nalit y is discussed in [5]. Our model is mor e expr essiv e – the exa mple we discuss in this pa per cannot b e des cribed by the mo del in [5]. It is also, in our view more natura l, and mathematically more elega n t. W e will make some comparison with the cited models in the last section o f the paper . The idea of [4] was to in tro duce t wo-sided automata, in or der to per m it op- erations analo gous to the pa rallel, ser ies a nd feedback o f classical circuits. F or techn ical rea sons which will b ecome clear we firs t int ro duce weigh ted automa ta (where the weigh ting of a transition is a non-negativ e real num b er) and then Marko v automata. W e then sho w ho w to comp ose suc h automata, ca lculat- ing the pro babilities in comp osed systems. An imp ortant asp ect is the us e of 1 conditional pr obabilit y since, for example, comp osing intro duces restrictions on po ssible transitions and hence changes probabilities. As an illustration of the algebra w e show how to specify a system of n dining philoso phers (a system with 12 n states) a nd to calculate the probability of reaching deadlo ck in k s t eps, a nd we show that this probability tends to 1 as k tends to ∞ , using the metho ds o f Perro n-F rob enius theory . It is clear that the algebra extends to semirings other than the re al num b ers, and in a later work we in tend to discuss examples such as quantum automata. An ea rlier v ers ion o f this pa per was presented at [3]. W e are grateful for helpful comments b y Pa we l Sob oci ´ nski and Ruggero Lanotte. 2 Mark o v automata Notice that in order to conserve symbo ls in the following definitions w e shall use the s ame symbol for the automa ton, its state space and its family o f matrices of transitions, distinguishing the separate pa rts only by the font. Definition 2. 1 Consider two finite al phab ets A and B , c ontaining, r esp e ctively, the symb ols ε A and ε B . A w eighted automaton Q with left in terface A and r igh t int erfac e B c onsists of a fin i te set Q of states, and an A × B indexe d family Q = ( Q a,b ) ( a ∈ A,b ∈ B ) of Q × Q matric es with non-ne gative r e al c o efficients. We denote the elements of the matrix Q a,b by [ Q a,b ] q,q ′ ( q , q ′ ∈ Q ) . We re quir e fur- ther t h at the r ow su ms of the matrix Q ε A ,ε B (and henc e of Q = P a ∈ A,b ∈ B Q a,b ) ar e strictly p ositive. W e call the matrix Q = X a ∈ A,b ∈ B Q a,b . the t o tal matrix of the a u tomato n Q . Definition 2. 2 Consider two fin i te alphab ets A and B , c ontaining, resp e ctively, the s ymb ols ε A and ε B . A Markov automaton Q with left interface A a nd right int erfac e B , is a weighte d automaton satisfying the extr a c ondition that the r ow sums of the total matrix Q ar e al l 1 . That is, for al l q X q ′ X a ∈ A,b ∈ B [ Q a,b ] q,q ′ = 1 . We c al l [ Q a,b ] q,q ′ the probability of the tra nsition from q to q ′ with left signal a and rig h t signal b . The idea is that in a given state v arious tra nsitions to other states ar e possi- ble and oc cur with v ar ious probabilities, the sum o f these probabilities b eing 1 . The tr ansitions that o ccur hav e effects, which we may think of a signals , on the t wo in terface s of the automaton, w h ich signals ar e repr esen ted by letters in the alphab ets. It is fundament al n ot to think of the letters in A a nd B as inputs or outputs, but r ather signals induced by transitions of the automato n o n the int erfac es. F or examples see section 2 .3. When bo th A a nd B are one element sets a Mar k ov automaton is a Markov matrix. 2 Definition 2. 3 Consider a Markov automaton Q with interfac es A and B . A b ehaviour of length k of Q c onsists of a two wor ds of length k , one u = a 1 a 2 · · · a k in A ∗ and the other v = b 1 b 2 · · · b k in B ∗ and a se quenc e of non- ne gative r ow ve ctors x 0 , x 1 = x 0 Q a 1 ,b 1 , .x 2 = x 1 Q a 2 ,b 2 , · · · , x k = x k − 1 Q a k ,b k . Notic e that, in gener al, x i is not a distribution of states; fo r example, in our examples often x i = 0 . There is a str aigh tforward wa y of conv erting a weighted automaton into a Marko v automaton which we c all normalization . 2.1 Normalization Definition 2. 4 The normalizatio n of a weighte d automaton Q , denote d N ( Q ) is the Markov automaton with the same interfac es and st at es, but with h N ( Q ) a,b i q,q ′ = [ Q a,b ] q,q ′ P q ′ ∈ Q [ Q ] q,q ′ = [ Q a,b ] q,q ′ P q ′ ∈ Q P a ∈ A,b ∈ B [ Q a,b ] q,q ′ . T o see that N ( Q ) is Mar k ov, notice that the q th row sum of N ( Q ) is X q ′ X a,b h N ( Q ) a,b i q,q ′ = X q ′ X a,b " [ Q a,b ] q,q ′ P P a,b [ Q a,b ] q,q ′ # q,q ′ = P q ′ P a,b [ Q a,b ] q,q ′ P q ′ P a,b [ Q a,b ] q,q ′ = 1 . Lemma 2. 5 (i) I f Q is a Markov automaton then N ( Q ) = Q . (ii) If c a,b,q ar e p ositive r e al numb ers and Q and R ar e weighte d aut o mata (with the same interfac es A and B , and t h e same st a te sp ac es Q = R ) such that [ Q a,b ] q,q ′ = c q [ R a,b ] q,q ′ then N ( Q ) = N ( R ) . Pro of. ( ii) follows since X q ′ X a,b [ Q a,b ] q,q ′ = X q ′ X a,b c q [ R a,b ] q,q ′ = c q X q ′ X a,b [ R a,b ] q,q ′ and hence [ Q a,b ] q,q ′ P q ′ P a,b [ Q a,b ] q,q ′ = c q [ R a,b ] q,q ′ c q P q ′ P a,b [ R a,b ] q,q ′ = [ R a,b ] q,q ′ P q ′ P a,b [ R a,b ] q,q ′ .  An imp ortant ope ration o n weigh ted auto m ata is the p ower constr uction. 3 2.2 The p o wer construction Definition 2. 6 If Q is a weighte d automaton and k is a natu r al numb er, then form a weighte d aut o maton Q k as fol lows: the states of Q k ar e those of Q ; the left and right interfac es ar e A k and B k r esp e ctively; ε A k = ( ε A , · · · , ε A ) , ε B k = ( ε B , · · · , ε B ) . If u = ( a 1 , a 2 , · · · , a k ) ∈ A k and v = ( b 1 , b 2 , · · · , b k ) ∈ B k then ( Q k ) u,v = Q a 1 ,b 1 Q a 2 ,b 2 · · · Q a k ,b k . If Q is weigh ted and u = ( a 1 , a 2 , · · · , a k ) ∈ A k , v = ( b 1 , b 2 , · · · , b k ) ∈ B k , then [( Q k ) u,v ] q,q ′ is the sum ov er all paths from q to q ′ with left sig nal sequence u a nd righ t signal sequence v o f t he weigh ts of paths, where the weigh t of a path is the pro duct of the w eights of the steps. Lemma 2. 7 If Q is a weighte d automaton t hen t h e total matrix of Q k is t h e matrix p ower Q k . Henc e if Q is Markov then so is Q k . Pro of. The q , q ′ ent ry of the total matrix of Q k is X u ∈ A k ,v ∈ B k  ( Q k ) u,v  q,q ′ = X u ∈ A k ,v ∈ B k [ Q a 1 ,b 1 Q a 2 ,b 2 · · · Q a k ,b k ] q,q ′ = X u ∈ A k ,v ∈ B k X q 1 , ··· q k − 1 [ Q a 1 ,b 1 ] q,q 1 [ Q a 2 ,b 2 ] q 1 ,q 2 · · · [ Q a k ,b k ] q k − 1 ,q ′ = X q 1 , ··· q k − 1 X a 1 , ··· ,a k X b 1 , ··· ,b k [ Q a 1 ,b 1 ] q,q 1 [ Q a 2 ,b 2 ] q 1 ,q 2 · · · [ Q a k ,b k ] q k − 1 ,q ′ = X q 1 , ··· q k − 1 X a 1 , b 1 [ Q a 1 ,b 1 ] q,q 1 X a 2 , b 2 [ Q a 2 ,b 2 ] q 1 ,q 2 · · · X a k ,b k [ Q a k ,b k ] q k − 1 ,q ′ = [ QQ · · · Q ] q,q ′ .  Definition 2. 8 If Q is a Markov aut o maton then we c al l Q k the automaton of k step paths in Q . We define the pr ob ability in Q of p assing fr om state q to q ′ in exactly k st ep s with left signal u and right signal v t o b e [( Q k ) u,v ] q,q ′ . It is important to unders t and the precise m eaning of this definition. The probability of pa ssing from state q to q ′ in pr ecisely k steps, so defined, is the weigh ted prop ortion o f a ll paths of length k beg inning at q and ending at q ′ amongst all paths of pr ecisely length n be ginning at q . 2.3 Graphical represen tation Although the definitions ab o ve are mathematically straightforward, in practice a gra ph ical notation is mor e intuitiv e. W e may compres s the description o f an automaton with in terfaces A and B , which requires A × B matrices, into a single lab elled graph, like the ones intro duced in [4]. F urther, express ions of automata in this alg ebra may be drawn as “circuit diag rams” also as in [4]. W e indicate bo th of these matters b y describing some examples . 4 2.3.1 A philo sopher Consider the alphab et A = { t, r, ε } . A philo sopher is an automaton Phi l with left in terface A and r igh t interfaces A , state space { 1 , 2 , 3 , 4 } , and transition matrices Phil ε,ε =     1 2 0 0 0 0 1 2 0 0 0 0 1 2 0 0 0 0 1 2     , Phil t,ε =     0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0     , Phil ε,t =     0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0     Phil r,ε =     0 0 0 0 0 0 0 0 0 0 0 1 2 0 0 0 0     , Phil ε,r =     0 0 0 0 0 0 0 0 0 0 0 0 1 2 0 0 0     . The other four tr ansition matrices are z ero ma t rice s. Notice that the total matrix of Phil is     1 2 1 2 0 0 0 1 2 1 2 0 0 0 1 2 1 2 1 2 0 0 1 2     , which is clea rly s tochastic, so Phil is a Marko v automaton. The inten tion b ehind these matrices is a s f ollows: in all states the philos opher do es a transition la belled ε, ε ( id le tr ansition ) with probability 1 2 ; in state 1 he do es a transition to state 2 with pro b ability 1 2 lab elled t, ε ( take the left fork ); in state 2 he do es a transition to state 3 with proba bilit y 1 2 lab elled ε, t ( take t h e right fork ); in state 3 he do es a transition to state 4 with probability 1 2 lab elled r , ε ( r ele ase the left fork ); and in state 4 he do es a transition to state 1 with probability 1 2 lab elled ε, r ( r ele ase the left fork ). All this information may b e put in the following diagram. ✲ ❄ ✛ ✻ ✰ ❃ ■ ❘ 1 2 3 4 ε, ε ; 1 2 ε, ε ; 1 2 ε, ε ; 1 2 ε, ε, 1 2 t, ε ; 1 2 ε, t ; 1 2 r , ε ; 1 2 ε, r ; 1 2 t, r , ε t, r , ε 5 2.3.2 A fork Consider again the alphab et A = { t, r, ε } . A fork is an a utomaton F o r k with left interface A and right in terface A , state space { 1 , 2 , 3 } , and transitio n matrices F ork ε,ε =   1 3 0 0 0 1 2 0 0 0 1 2   , F ork t,ε =   0 1 3 0 0 0 0 0 0 0   , F ork ε,t =   0 0 1 3 0 0 0 0 0 0   F ork r,ε =   0 0 0 1 2 0 0 0 0 0   , F ork ε,r =   0 0 0 0 0 0 1 2 0 0   . The other four transition matrices a re ze ro. F ork is a Marko v automaton since its total matrix is   1 3 1 3 1 3 1 2 1 2 0 1 2 0 1 2   . The in tention behind these matrices is as follo ws: in all states the fork do es a transition lab elled ε, ε ( id le tr ansition ) wit h p ositive probability (either 1 3 or 1 2 ); in state 1 it do es a tr ansition to state 2 with pr obabilit y 1 3 lab elled t, ε ( taken to the left ); in state 1 he do es a transition to state 3 with probability 1 3 lab elled ε, t ( taken to the right ); in s t ate 2 he do es a transition to state 1 with probability 1 2 lab elled r , ε ( r ele ase d t o the left ); in state 3 he do es a tra n sition to state 1 with probability 1 2 lab elled ε, r ( r ele ase d to t h e right ). All this informatio n may b e put in the following diagr am: 1 2 3 ❘ ■ ✠ ✒ ✒ ■ ✠ ε, ε ; 1 3 ε, ε ; 1 2 ε, ε ; 1 2 t, ε ; 1 3 ε, t ; 1 3 r , ε ; 1 2 ε, r ; 1 2 2.4 Reac habilit y F or many applications we are interested only in s t ates reachable fro m a given initial state by a path of p ositive pro babilit y . Given a Marko v a utomaton Q and an initial state q 0 there is a subauto maton Reac h ( Q ,q 0 ) whose states a re the reachable states, and who se transitions are th os e of Q r estricted to t he reachable states. 6 3 The algebra of Mark o v automata: op erations Now we define op erations o n weigh ted automa t a a nalogous (in a pr ecise sense) to those defined in [4]. Definition 3. 1 Given weighte d automata Q with left and right i nterfac es A and B , and S with interfac es C and D t he parallel compos it e Q × R is the weighte d automaton which has set of s t ates Q × R , left interfac es A × C , right interfac e B × D , ε A × C = ( ε A , ε C ) , ε B × D = ( ε B , ε D ) and tra nsition matric es ( Q × S ) ( a,c ) , ( b,d ) = Q a,b ⊗ S c,d . This just says that the w eight o f a transition from ( q , r ) to ( q ′ , r ′ ) with left signal ( a, c ) and r igh t signal ( b, d ) is the pro duct of the weigh ts of the transitio n q → q ′ with sig nals a and b , and the weigh t of the transitio n r → r ′ with sig nals c and d . Lemma 3. 2 If Q and R ar e weighte d automata then N ( Q × R ) = N ( Q ) × N ( R ) . Henc e if Q and R ar e Markov automata then so is Q × R . Pro of. h N ( Q × R ) ( a,c ) , ( b,d ) i ( q,r ) , ( q ′ ,r ′ ) = [ Q a,b ] q,q ′ [ S c,d ] r,r ′ P q ′ ,r ′ P ( a,c ) , ( b,d ) [ Q a,b ] q,q ′ [ S c,d ] r,r ′ = [ Q a,b ] q,q ′ [ S c,d ] r,r ′ P q ′ , ( a,b ) [ Q a,b ] q,q ′ P r ′ , ( c,d ) [ S c,d ] r,r ′ = h N ( Q ) ( a,b ) i q,q ′ h N ( R ) ( c,d ) i r,r ′ = h ( N ( Q ) × N ( R )) ( a,c ) , ( b,d ) i ( q,r ) , ( q ′ ,r ′ ) . F or the second par t no tice that if Q and R are Marko v then Q × R = N ( Q ) × N ( R ) = N ( Q × R ) which implies that Q × R is Mar k ov.  Definition 3. 3 Given weighte d automata Q with left and right i nterfac es A and B , and R with interfac es B and C the series (comm unicating pa rallel) comp osite of weigh ted automata Q ◦ R ha s set of states Q × R , left interfac es A , right int erf ac e C , and tr ansition matric es ( Q ◦ R ) a,c = X b ∈ B Q a,b ⊗ R b,c . Lemma 3. 4 ( Q ◦ R ) ◦ S = Q ◦ ( R ◦ S ) . 7 Pro of. This follows from the fact that ⊗ is asso ciativ e..  It is e asy to see that Q ◦ R is no t necessa rily Markov ev en when b oth Q a nd R are. The rea son is that the communication in the s eries comp osite reduces the n um b er of poss ible transitions, so that we must normalize to get (co ndi- tional) probabilities. Howev er in a multiple comp osition it is only necessa ry to normalize a t the e nd , b ecause of the following lemma. Lemma 3. 5 N ( N ( Q ) ◦ N ( R )) = N ( Q ◦ R ) . Pro of. h ( NQ ◦ NR ) a,c i ( q,r ) , ( q ′ ,r ′ ) = X b ∈ B  NQ a,b  q,q ′ ⊗ [ NR b,c ] r,r ′ = X b ∈ B [ Q a,b ] q,q ′ P q ′ P a,b [ Q a,b ] q,q ′ · [ R b,c ] r,r ′ P r ′ P b,c [ R b,c ] r,r ′ ′ = 1 P q ′ ,r ′ ( P a,b [ Q a,b ] q,q ′ P b,c [ R b,c ] r,r ′ ) X b ∈ B [ Q a,b ] q,q ′ [ R b,c ] r,r ′ . = c q,r h ( Q ◦ R ) a,c i ( q,r ) , ( q ′ ,r ′ ) , where c q,r = 1 P q ′ ,r ′ ( P a,b [ Q a,b ] q,q ′ P b,c [ R b,c ] r,r ′ ′ ) depe n ds only on q , r . Hence by the lemma 2 .5 ab o ve N ( NQ ◦ NR ) = N ( Q ◦ R ) .  Definition 3. 6 If Q and R ar e Markov automata, Q with left interfac e A and right interfac e B , R with left interfac e B and right interfac e C then the s eries c omp osite of Markov aut o mata Q · R is define d t o b e Q · R = N ( Q ◦ R ) . The or em 3.7 ( Q · R ) · S = Q · ( R · S ) . Pro of. ( Q · R ) · S = N ( N ( Q ◦ R ) ◦ S ) = N ( N ( Q ◦ R ) ◦ N ( S )) s ince S is Mar k ov = N (( Q ◦ R ) ◦ S ) = N ( Q ◦ ( R ◦ S )) b y 3.6 and 3.6 = N ( N ( Q ) ◦ N ( R ◦ S )) by 3.6 = N ( Q ◦ N ( R ◦ S )) = Q · ( R · S ) since Q is Marko v .  Theorem 3.8 If Q and R ar e Markov automata then ( Q × R ) k = Q k × R k 8 Pro of. ( Q × R ) k ) ( u,u ′ ) , ( v ,v ′ ) = ( Q a 1 ,b 1 ⊗ R a ′ 1 ,b ′ 1 )( Q a 2 ,b 2 ⊗ R a ′ 2 ,b ′ 2 ) · · · ( Q a k ,b k ⊗ R a ′ k ,b ′ k ) = ( Q a 1 ,b 1 ⊗ Q a 2 ,b 2 ⊗ · · · Q a k ,b k )( R a ′ 1 ,b ′ 1 ⊗ R a ′ 2 ,b ′ 2 · · · ⊗ R a ′ k ,b ′ k ) = ( Q k × R k ) ( u,u ′ ) , ( v ,v ′ ) .  Remark. It is not the case that if Q , R are Ma rk ov then ( Q · R ) k = Q k · R k . The reaso n is that nor m alizing length k steps in a weigh ted automaton is not the s ame as co nsidering k step paths in the normaliza tion of the a ut omato n. Next we define some constants ea c h of which is a Marko v automaton. Definition 3. 9 Given a r elation ρ ⊂ A × B such that ( ε A , ε B ) ∈ ρ we define a Markov aut omaton ρ as fol lows: it has one state ∗ say. The t r ansition matric es [ ρ a,b ] ar e 1 × 1 matric es, that is, r e al numb ers. L et | ρ | b e the nu m b er of elements in ρ . Then ρ a,b = 1 | ρ | if ρ r elates a and b , and ρ a,b = 0 otherwise. Some sp ecial ca ses, all descr ibed in [4], hav e particular impor tance: (i) the a ut omato n cor responding to the identit y function 1 A , consider ed as a relation on A × A is ca lled 1 A ; (ii) the automaton corre sponding to the dia gonal function ∆ : A → A × A (considered as a rela tion) is ca lled ∆ A ; the a u tomato n corres ponding to the o pposite r elation o f ∆ is called ∇ A . (iii) the a u tomato n corresp onding to the function twis t : A × B → B × A is called twist A,B . (iv) the automaton cor responding to the rela tion η = { ( ∗ , ( a, a ); a ∈ A } ⊂ {∗} × ( A × A ) is called η A ; the automa t on corr esponding to the opp osite of η is called ǫ A . 3.1 The dining philosophers system Now the mo del of the dining philosophers pr oblem we co nsider is an expre ssion in the algebra , inv olving a lso the automa ta Phil and F ork . The the sys t em of n dining philosopher s is DF n = η A · (( Phil · F ork · Phil · F ork · · · · · Phil · F ork ) × 1 A ) · ǫ A , where in this expressio n there are n philosophers and n fork s. As explained in [4], w e may represent this system by the following diagr am, where we abbr eviate Phil to P and F o r k to F . P F P F P F · · · 9 Let us examine the c ase when n = 2 with initial state (1 , 1 , 1 , 1 ). Le t Q be the reachable part of DF 2 . The states rea c hable from the initial sta te ar e q 1 = (1 , 1 , 1 , 1 ) , q 2 = (1 , 3 , 3 , 2), q 3 = (3 , 2 , 1 , 3 ) , q 4 = (1 , 1 , 4 , 2 ) , q 5 = (4 , 2 , 1 , 1 ) , q 6 = (1 , 3 , 2 , 1 ) , q 7 = (2 , 1 , 1 , 3 ) , q 8 = (2 , 3 , 2 , 3 ) ( q 8 is the unique deadlo c k s tate). The single matrix of the automaton Q , us ing this o rdering of the states, is             1 4 0 0 0 0 1 4 1 4 1 4 0 1 2 0 1 2 0 0 0 0 0 0 1 2 0 1 2 0 0 0 1 2 0 0 1 2 0 0 0 0 1 2 0 0 0 1 2 0 0 0 0 1 3 0 0 0 1 3 0 1 3 0 0 1 3 0 0 0 1 3 1 3 0 0 0 0 0 0 0 1             . Calculating pow ers of this matrix we s ee that the probability of reaching deadlo c k fro m the initial sta te in 2 s t eps is 23 48 , in 3 steps is 341 576 , and in 4 steps is 4415 6912 . 4 The probabilit y of deadlo c k The idea of this section is to apply Perro n-F rob enius theory (see, for example) [7] to the Dining Philosopher automaton. Ho wev er, for conv enience, we give the details of the pro of of the case we need, without refering to the general theorem. Definition 4. 1 Consider a Markov automaton Q with input a nd out p ut sets b eing one element sets { ε } . A state q is c al le d a deadlock if the only t r ansition out of q with p ositive pr ob ability is a t r ansition fr om q to q (t h e pr ob ability of the tr ansition mu st ne c essarily b e 1 ). Theorem 4.2 Consider a Markov automaton Q with interfac es b eing one ele- ment sets, with an initial state q 0 . Supp ose that (i) Q has pr e cisely one r e achable de ad lo ck state, (ii) f or e ach r e achable st ate, not a d e ad lo ck, ther e is a p ath with non-zer o pr ob ability t o q 0 , and (iii) for e ach r e achable state q t h er e is a t r ansition with non- zer o pr ob ability to itself. Then the pr ob ability of re aching a de ad lo ck fr om the initial state in k steps tends to 1 as k tends to infi n i ty. Pro of. Let R = Reac h ( Q , q 0 ) . Suppo se R has m states. Then in writing the matrix R we choos e to put the deadlo ck la st, so that R has the form R =  S T 0 1  where S is ( m − 1) × ( m − 1) and T is ( m − 1) × 1. Now R k =  S k T k 0 1  10 for so me matrix T k . Condition (i) implies tha t ther e is a path with p ositive probability (a positive path) from any non-dea dlock s tate to an y other in R . Condition (ii) implies tha t if there is a p ositive path of length l b et ween t wo states then there is a lso a p ositive path o f all lengths greater tha n l . The se tw o facts imply tha t there is a k 0 such that from an y no n-deadlock state to an y other state there is a positive path o f length k 0 . F or this k 0 the matrix T k 0 is strictly po sitiv e . This means that the row sums o f S k 0 are strictly less than 1. But the eigenv alues of a matrix are dominated in abso lute v alue by the maximum of the absolute row sums (the sums of absolute v alues of the row elemen ts). Hence the eigenv alues of S k 0 and hence of S all hav e abso lut e v alue les s than 1. But by considering the Jorda n cano nical for m of a matrix whose eigenv alues all hav e absolute v alues less than 1 it is ea sy to see that S k tends to 0 as k tends to infinit y . Hence T k tends to the co lum n vector a ll o f whose entries are 1. Hence the proba bilit y of reaching the deadlo c k fro m any of the other states in k steps tends to 1 as k tends to infinity .  Corollary 4. 3 In the dining philosopher pr oblem DF n with q 0 b eing the state (1 , 1 , · · · , 1 ) the pr ob ability of r e aching a de ad lo ck fr om the initial state in k steps tends t o 1 as k t ends to infinity. Pro of. W e just n eed to verify the conditions of the theor em for the dining philosopher pr oblem. It is str aigh tforward to chec k that the state (2 , 3 , 2 , 3 , · · · , 2 , 3) in which t he philoso phers ar e all in state 2 and the forks in state 3 is a reachable deadlo c k.It is clear that in a n y state q there is a p ositiv e tra nsition to q , since each comp onen t has silen t moves in each s tate. W e need only chec k that for any reachable state other than this deadlo ck that there is a p ositiv e pa th to the initial s tate. Consider the states f 1 , f 2 of tw o forks a dj acent mo dulo n , and the s tate p o f the philo sopher betw een these tw o forks. Examining the p ositiv e paths p ossible in tw o adjacent forks and the cor responding philosopher we see that the r eac hable config urations are limited to (a) f 1 = 1, p = 1, f 2 = 1, (b) f 1 = 1, p = 1, f 2 = 3, (c) f 1 = 1, p = 4, f 2 = 2, (d) f 1 = 2, p = 1, f 2 = 1, (e) f 1 = 2, p = 1, f 2 = 3, (f ) f 1 = 3, p = 2, f 2 = 1, (g) f 1 = 3, p = 2, f 2 = 3, (h) f 1 = 3, p = 3 , f 2 = 2, (i) f 1 = 2, p = 4 , f 2 = 2. W e will show that in states other tha n the dea dlock or the inital s t ate there is a tra nsition o f the sy stem which increa ses the num b er of forks in state 1. Notice that in a reachable s tate the sta t es of adjacent forks determine the state o f the philosopher betw een. Consider the p ossible configurations of for k s tates. W e need no t consider ca ses all forks are in s tate 1 (initial), or a ll in state 3 (the known deadlo c k). Giv en t wo a dj acent for ks in states 3 , 2 there a re transitio ns which only inv olve this philosopher a nd the tw o for ks (apar t from null signals) which result in one o f the forks retur ning to state 1 (the philosopher puts down a fork that he holds). This is also the cas e when tw o adjacent forks are in sta tes 1 , 2 or 2 , 2 or 3 , 1. But in a circular arra ngemen t other than all 1’s or a ll 3’s one of the pairs 1 , 2 or 2 , 2 or 3 , 1 o r 3 , 2 must o ccur.  Remark. Notice that in the pro of of the cor ollary we did not use the sp ecific po sitiv e proba bilities of the actions of the philosopher s and for ks. Hence the 11 result is true with any p ositive proba bilities re placing the sp ecific ones we gav e in the description of the p hiloso pher and f or k. In fact, different philo sophers and fork s may hav e different proba bilit ies without affecting the conclusion of the c orollary . 5 Concluding remarks 5.1 The algebra of automata: equations There is muc h more to say ab out the algebra ic structur e and its rela t ions with other fields . W e have mentioned a bov e some eq u atio n s whic h are sa tisfied, and here we mention one more. Lemma 5. 1 The c onstants ∆ A , ∇ A satisfy the F r ob en i us e quations [1], n amely that ( ∆ A × 1 A ) · ( 1 A × ∇ A ) = ∇ A · ∆ A . The pro of is straig h tforward. 5.2 Comparisons According to Rabin a pr ob abilistic automaton o n an alpha bet Σ co nsists of a set of states Q a nd a family of sto c hastic transition matr ices [ P a ] q,q ′ ( a ∈ Σ; q , q ′ ∈ Q ). A distribution of states is a r o w v ector with non negative real entries whose sum is 1. A b eha viour corres ponding to an initial state distribution x 0 , and an input w or d u = a 1 a 2 · · · a k , is a sequence of state distributions x 0 , x 1 = x 0 P a 1 , x 2 = x 0 P a 2 , · · · , x k = x k − 1 P a k . This is a non-comp ositional mo del, and it immediately clear t hat the meaning of the alpha bet in Rabin is quite differe n t from the meaning of the alphab ets for our Marko v automata. F or Rabin the letters are inputs w hic h dr iv e the automa ton Q – for a g iv en sta te q and a giv en input a the sum of the pr obabilities of transitions out of q is 1. W e a re a ble to des cribe the same phenomeno n by considering a second automa ton R whose signal o n the in terface drive Q , which of cour se in tro duces conditional probabilities . F rom our p oin t of v iew Rabin’s probabilities ar e conditional ones resulting from the knowledge that an input a o ccurs. Another difference is that every tra nsition in every state in Rabin’s automata pro duces a distribution of states. How ever it is crucial in o ur mo del that actions are not necessa rily defined in a ll states; or if they are de fi ned they may be only partially defined. F or example, the fork in state 2 has no trans itions lab elled t, ε ; it cannot b e tak en aga in when it is a lready tak en. The fork in state 1 may b e taken to the left with probability 1 2 . The second mo del we mentioned [5] co nsiders a ge neralization of Rabin’s mo del (and hence differen t fro m ours) which is influenced b y co ncurrency theory . It has a form of comp osition, and as usual with mo dels related to pro cess a lge- bras the comp osition inv olves an underlying bro adcast (and hence interlea ved) communication, a nd do es not inv olve conditiona l probability . As a result it is not p ossible to descr ibe our example, in which, in a s ingle step a ll philoso - phers may ta k e their left fork. Instea d, it is straigh tforward to model br oadcast, int erleaved mo dels using our algebra [2], using in pa rticular the co mponent ∆. 12 F urther, [5] has a muc h more limited alge bra than that presented here; for example, multiply simultaneous sig nals (to gether with the synchronization on some o f the signals) are not av ailable. In our view interleaving destroys the realism of the mo del. F or example, to reach deadlock in the dining philosopher problem re quires a seq uence of actions, as philosopher s take the forks one b y one. This r esults in quite different probabilities. References [1] A. Carb oni, R.F.C. W a lt ers , Cartesian bic ate gories I , Journal of Pure and Applied Alge bra, 49, 11 –32, 1987. [2] L. de F rances co Albasini, N. Sa badini, R.F.C. W alter s, The p ar al lel c om- p osition of pr o c esses , AR T 2008, Analysing Reduction systems using T ran- sition sy stems, 111–12 1, F o rum, Udine, 2 008. [3] L. de F r ancesco Albasini, N . Sa badini, R.F.C. W alters: Cosp an Sp an(Gr aphs): a c omp ositional mo del for r e c onfigur able automata nets, De- velopmen ts and New T racks in T race Theory , Cr emona, Italy , 9-11 Octob er 2008. [4] P . Katis, N. Sabadini, R.F.C. W alter s, Sp an(Gr aph): A c ate goric al alge- br a of tr ansition s yst ems , Pr oc. AMAST ’97, SLNCS 1349 , pp 30 7–321, Springer V erlag, 1997. [5] Nancy A. Lynch , Ro berto Segala, F r its W. V aandrag er, Comp ositionality for Pr ob abilistic Automata , P roc. CONCUR 200 3, Springer Lecture Notes in C omputer Science, 27 61, pp 204 -222, 2 003. [6] M. O Ra bin, Pr ob abilistic Automata , Information and Control 6 , pp.23 0- 245, 1 963. [7] C. Go dsil and G. Royle, Alg ebr aic Gr aph The ory , Springer, 2001 13