Additivity of on-line decision complexity is violated by a linear term in the length of a binary string

Additivit y of on-line decision complexit y is violated b y a linear term in the l ength of a binary s tring DRAFT Bruno Bau wens ∗ Octob er 31, 2018 Abstract W e sho w that there are in finitely m any binary strings z , su ch that the sum of the on-line decision complexit y of pr edicting the ev en bits of z giv en the previous u n ev en bits, and the d ecision complexit y of predicting t he un ev en bits gi ven the previous ev en t bits, exceeds the Kolmogoro v complexit y of z by a linear term in the length of z . Keywords: Decision complexit y – Kolmogoro v complexit y – De- comp ositions of Kolmogoro v complexit y 1 In tro duction On-line decision complexit y has b een intro duced and inv estigated in [ ? , ? ]. It also naturally app ears in the definition of ideal influence tests [ ? , ? ]. A natural question is whether algorithmic m utual information of t w o time series x, y , can b e decomp osed into an info rmation flow going f rom x to y , a flo w going from y to x , and an information flo w instan taneously presen t in b oth strings. It turns out [ ? ] that this question is related t o the question of defining a decomp o sition of K ( x, y ) with l ( x ) = l ( y ) as the sum of the complexit y of predicting x i +1 giv en x 1 ...x i and y 1 ...y i , i 6 n , and the complex ity of predicting y i +1 giv en x 1 ...x i +1 and y 1 ...y i . It will b e shown that using on-line decision complexit y for this complexit y , ∗ The re s ult and its motiv ation was presented at 200 9 conference of Log ic, Com- putabilit y and Randomnes s in Lumin y [ ? ]. Depar tment of Electrical Energ y , Systems and Automation, Ghent University , T ec hnolog ie park 9 1 3, B -9052 , Ghent, Belgium, Bruno.Bauw ens@ ugent.be. Supp orted b y a Ph.D grant of the Institute for the Pro mo- tion of Innov a tion thro ugh Science and T ec hnolog y in Flanders (IWT-Vlaanderen). 1 this sum exceeds K ( x, y ) by a linear constant in l ( x ). A mo dification of this definition of on-line decision complexit y will b e sho wn to hav e an appro ximate decomp osition [ ? , ? ]. Non-additivit y of decision complexit y was also show n in [ ? ], in the con text of randomness defined b y sup ermartingales. Using nat ura l def- initions for randomness a parado x is sho wn: if the ev en bits o f z giv en the past unev en bits of z are random, and also t he unev en bits o f z give n the past ev en bits of z are random, than it is p ossible tha t z is not ran- dom. The pro of of this result implies that a dditivity of on-line decision complexit y is violated b y a log arithmic term. 2 Definiti o ns and notation F or excellen t intro ductions to Kolmogorov complexit y w e r efer to [ ? , ? ]. Let ω , ω <ω , 2 N and 2 <ω denote the set of the Natural n um b ers, the set of finite sequenc es of Natural n um b ers, the binary strings o f length N , and the binary strings of finite length. Other definitions are analogue. Let ǫ denote t he empty sequence. Remark that there is a natural bijection b et we en ω and 2 <ω , defined b y: ǫ → 0 , 0 → 1 , 1 → 2 , 00 → 3 , 01 → 4 , ... [ ω ] is the set of nested sequences of Natural n um b ers, with finite depth. Mathematically , it is the closure of ω under the mapping f ( S ) = S <ω . Remark that there is a computable bijection b et we en ω a nd [ ω ], t herefore most complexit y and computabilit y results in ω also hold in [ ω ]. An in terpreter Φ is a pa rtial computable function from 2 <ω × [ ω ] → [ ω ]. An in terpreter is prefix-free if for any x , t he set D x of all p where Φ( p | x ) is defined, is prefix-free. Let Φ b e some fixed optimal univ ersal prefix-free interpreter. F or any x ∈ 2 <ω , l ( x ) denotes t he length of x . F or any x ∈ ω <ω , l ( x ) corresp onds to the length of some prefix-free enco ding of x on a binar y tap e: l ( x ) = l ( x ) X i =1 2 log x i . F or x, y ∈ [ ω ], the Kolmo gor ov c omplexity K ( x | y ), is defined as: K ( x | y ) = min { l ( p ) : Φ( p | y ) ↓ = x } . The Kolmog oro v complexit y of elemen ts in 2 <ω is defined b y using the computable bijection men tioned in the b eginning of t his section. 2 F or Z ∈ [ ω ] , Q, A ∈ ω n , Q i denotes Q 1 ...Q i . The on-line decision com- plexit y is defined b y: K ( Q 1 → A 1 ; ... ; Q n → A n | Z ) = min { l ( p ) : ∀ i < n [Φ( p | Q i , Z ) ↓ = A i ] } . This definition differs sligh tly with the definition of [ ? ], with resp ect that A ∈ ω n is c hosen, in stead of A ∈ 2 n . Also a shorter notation [ ? ] will b e used: K ( x | y ↑ ) = K (0 → x 1 ; ... ; y n − 1 → x n ) , K ( y | x ↑ + ) = K ( x 1 → y 1 ; ... ; x n → y n ) . 3 Main res ult and pro of tactic Prop osition 3.1. ∃ c > 0 ∃ ∞ x, y ∈ ω <ω  K ( x | y ↑ ) + K ( y | x ↑ + ) − K ( x, y ) > c ( l ( x ) + l ( y ))  . In [ ? ] a nd rep eated in [ ? , ? ], it is prov en that fo r an y n there is an x ∈ 2 n suc h that: K ( K ( x ) | x ) > + log n − log log n. Let y b e the binary expansion of K ( x ). F rom this a nd equation (4) it can b e shown that K ( x ) + K ( y | x ) − K ( x, y ) > + log n − log log n. By inserting zeros at the right places in x, y , it can b e sho wn that there exists infinit y many x, y with l ( x ) = l ( y ): K ( x | y ↑ ) + K ( y | x ↑ + ) − K ( x, y ) > O (log l ( x )) . This sho ws prop osition 3.1 for a loga rithmic term in l ( x ). It seems nat- ural to think that such a result can be impro v ed to a linear term, b y concatenating suc h strings. This is what ev en tually will happ en in the pro of, at equation (12). Ho w ev er, to b e able to add up these differences, conditional complexitie s mu st add up in some w ay to on-line decision complexit y , in what extend this is p ossible is still an op en problem. Hap- pily , Lemma 4 .4 can circum v en t this, if some extra info rmation is av a il- able. This info rmation is stored in sequenc es u and v and is added to x and y . Adding this inf o rmation requires, some more b ounds to mak e the pro of work: (10), (11). The pro of b elow pro vides all tec hnical details. 3 4 Pro of First some definitions and lemmas are giv en. f ( x ) 6 + g ( x ) is short for f ( x ) 6 g ( x ) + O (1), and f ( x ) = + g ( x ) is short for f ( x ) = g ( x ) ± O (1) . F or any a, b ∈ [ ω ], a − → b means tha t there is a fixed p ∈ 2 <ω with l ( p ) 6 O (1 ), suc h that Φ( p | a ) ↓ = b . Remark that if a − → b , then K ( a ) > + K ( b ). The shortest program witnessing K ( a | b ) is denoted b y: a ∗ [ b ] = min { p : Φ( p | b ) ↓ = a } . a ∗ is short for a ∗ [ ǫ ]. R emark that: a ∗ [ b ] , b ← → ( a ∗ [ b ]) ∗ [ b ] , b. (1) Lemmas 4.1, 4.2, and 4.3 pro vide observ ations, kno wn within the com- m unit y , and stated here explicitly for later reference. Lemma 4.1. F or A ∈ ω <ω , X i 6 n K ( A i | A i − 1 ) > K ( A ) − O ( n ) . Pr o of. F or U, V ∈ ω , prefix-free complexit y satisfies additivity [ ? ]: K ( U, V | W ) = + K ( U | W ) + K ( V | U ∗ [ W ]) . (2) Since there is a computable bijection b et ween ω and [ ω ], this result also applies to [ ω ]. Let U, V ∈ [ ω ], since U ∗ [ W ] , W − → U , K ( V | U, W ) > + K ( V | U ∗ [ W ] , W ) . Inductiv e application of b oth equations ab ov e on A i pro v es the lemma. Lemma 4.2. F or a, b ∈ ω and c ∈ ω <ω : K ( a, b | c ) = + K ( a, b, K ( b | a ∗ [ c ] , c ) | c ) . Pr o of. The pro of b elow, sho ws the unconditioned v ersion o f the lemma, since the pro of of the conditioned vers ion is the same. In [ ? ] and exercise 3 . 3 . 7 in [ ? ] it is stated t ha t fo r ev ery w ∈ ω , and n > K ( w ): log |{ p ∈ 2 n : Φ( p ) ↓ = w }| 6 + n − K ( w , n ) , (3) and K ( w , K ( w )) = + K ( w ) . (4) 4 Therefore, for c constan t, there are an O (1) n umber of programs that compute a, b and ha ve length K ( a, b ) + c . Let S b e the set of these programs. Remark that the elemen ts of S can b e en umerated giv en a, b, K ( a, b ) a nd therefore, for any p ∈ S , using (4), w e hav e: K ( a, b ) = + K ( a, b, K ( a, b )) = + K ( p ) . (5) By equation ( 2), we hav e: K ( a, b ) = + K ( a ) + K ( b | a ∗ ) . The progra ms a ∗ and b ∗ [ a ∗ ], can b e combin ed into a program p computing a, b . This program p can be constructed suc h that p − → a ∗ , b ∗ [ a ∗ ], and it has a length b elo w K ( a, b ) + c , for c constan t and large enough. Therefore p ∈ S , and since b ∗ [ a ∗ ] − → K ( b | a ∗ ) = l ( b ∗ [ a ∗ ]): K ( p ) > + K ( a, b, K ( b | a ∗ )) . Com bining with equation (5), finishes the pro of. Lemma 4.3. F or b ∈ 2 <ω , a, c ∈ [ ω ] : K ( a, b | c ) > + K ( a, b ∗ [ a, c ] | c ) − 2 log l ( b ) . Pr o of. The unconditioned v ersion of the lemma is prov en, since the con- ditioned pro of is essen tially the same. It suffices to show that: K ( b ∗ [ a ] | a, b ) 6 + 2 log l ( b ) . Again the pro o f of the unconditioned v ersion of this equation is the same as the conditioned one: K ( b ∗ | b ) 6 + 2 log l ( b ) . Giv en b a nd K ( b ) all programs of length K ( b ) that output b can b e en u- merated. By equation (3) , there are maximally a constan t suc h programs, therefore: K ( b ∗ | b ) = + K ( K ( b ) | b ) . Remark that by the prefix-free co de b 1 0 b 2 0 ...b l ( b ) 1 w e hav e: K ( b ) 6 + 2 l ( b ) . Using the natural bijection b et w een ω and 2 <ω , this sho ws that fo r n ∈ ω , K ( n ) 6 + 2 log n . K ( K ( b ) | b ) 6 + K ( K ( b )) 6 + 2 log K ( b ) 6 + 2 log l ( b ) . 5 Let Z ∈ [ ω ], A, Q ∈ ω n for some n , and N ∈ ω . F or i < n , let T i = ( Q i | A i ) and T = ( T 1 , ..., T n ). K ( T ) = K ( A | Q ↑ , N ) K ( T i | Z ) = K ( A i | A i − 1 , Q i − 1 , N , Z ) . F or some fixed N , a nd f or all i 6 n , we define the sets S i and the n umbers L i : S 0 ( T ) = 2 N S i ( T ) = S i − 1 ∩ { p : Φ( p | Q i , N ) ↓ = A i } L i ( T ) = ( − 1 if | S i ( T ) | = 0 ⌈ log | S i ( T ) |⌉ otherw ise. A low er b ound for K ( T ) is now pro ven . Lemma 4.4. K ( T ) > min { N , X i K ( T i | L i − 1 ) − O ( n ) } . Pr o of. F or each i , a semimeasure P can b e constructed using A i − 1 , Q i , L i − 1 , N : P ( z ) = 2 − L i − 1 |{ p ∈ S i − 1 : Φ( p | A i − 1 , Q i , N ) ↓ = z }| . Remark that P defines a semimeasure and that P is en umerable. P ( A i ) = 0, for some i , implies that no pro gram of length N can solv e task T i , t hus K ( T ) > N . In this case the lemma is prov en. Assume | S i | > 1 and th us P ( A i ) > 0. By applying the co ding theorem [ ? ] on P , it follows t ha t: L i − 1 − L i > K ( T i | L i − 1 ) − O (1) . Summing ov er i , g iv es: L 0 − L n > X i K ( T i | L i − 1 ) − O ( n ) . (6) Let p b e a program of length K ( T ), solving ta sk T . It p ossible to a pp end 2 N − K ( T ) − O (1) differen t strings of length N − K ( T ) − O ( 1 ) to p , in o r der to obtain elemen ts f rom S n . Therefore: L n 6 + N − K ( T ) . (7) Observ e that L 0 = N . Com bining equations (6 ) and (7) pro v es the lemma. 6 Pr o of. of pr op o sition 3.1 . Let u, x, y , v ∈ ω n for some n . Let z = N , 0 , 0 , 0 , u 1 , x 1 , y 1 , v 1 , ..., u n , x n , y n , v n . Define: T ux,i = ( u i , x i | z 4 i ) T x,i = ( x i | z 4 i +1 ) T y v,i = ( y i , v i | z 4 i +2 ) . F or X = ux, y v , let D X, 1 = 0 and for i > 2 let: D X,i = L i − 1 ( T X ) − L i ( T X ) . Remark that: X j 6 i D X,j = N − L i ( X ) . Equations (8), (11), (10), and (12) are now derive d. • Let: u i = D ∗ y v,i − 1 [ z 4 i ] v i = D ux,i − 1 . A t the end of the pro of u, x, y , v , N will b e constructed suc h that equation (14) holds, and therefore, N > K ( T X ) − O ( n ) for X = ux, y v . Since z 4 i − → u i − → L i − 1 ( T ux ) z 4 i +2 − → v i − → L i − 1 ( T y v ) w e hav e by lemma 4.4: K ( T X ) > X i K ( T X,i ) − O ( n ) . (8) • Cho o se: y i = K ( T x,i ) ∗ [ z 4 i +2 ] . (9) By Lemma 4 .2, it follo ws that: K ( u i , x i | z 4 i ) = + K ( u i , x i , K ( x i | u ∗ i [ z 4 i ] , z 4 i ) | z 4 i ) . By equation (1), w e ha v e that u ∗ i [ z 4 i ] , z 4 i ← → u i , z 4 i , and therefore: K ( x i | u ∗ i [ z 4 i ] , z 4 i ) = + K ( x i | u i , z 4 i ) = K ( x i | z 4 i +1 ) = K ( T x,i ) 7 Therefore: K ( u i , x i , K ( x i | u ∗ i [ z 4 i ] , z 4 i ) | z 4 i ) = + K ( u i , x i , K ( T x,i ) | z 4 i ) . Remark that l ( x i ) = m , and therefore K ( T x,i ) 6 + 2 log m . By Lemma (4.3), w e ha v e: K ( u i , x i , K ( T x,i ) | z 4 i ) > K ( u i , x i , K ( T x,i ) ∗ [ z 4 i +2 ] | z 4 i ) − O (log log m ) . By definition o f y i , (9), this sho ws tha t: K ( u i , x i | z 4 i ) > K ( u i , x i , y i | z 4 i ) − O (log log m ) . (10) • F rom equations (1) and (9), w e ha v e: y i , z 4 i +2 ← → y ∗ i [ z 4 i +2 ] , z 4 i +2 . Therefore, K ( v i | z 4 i +3 ) = K ( v i | y i , z 4 i +2 ) = + K ( v i | y ∗ i [ z 4 i +2 ] , z 4 i +2 ) = + K ( y i , v i | z 4 i +2 ) − K ( y i | z 4 i +2 ) (11) • In [ ? , ? , ? ] it is sho wn that for all m, w there is an x ∈ 2 m suc h that K ( K ( x | w ) | x, w ) > log m − log log m − O (1) . Actually , the unconditioned v ersion is sho wn, but this v ersion has the same pro of. Fix an m large enough and choo se x i ∈ 2 m suc h that b y equation (9): K ( y i | z 4 i +2 ) = K ( K ( T x,i ) ∗ [ z 4 i +2 ] | z 4 i +2 ) > + K ( K ( T x,i ) | z 4 i +2 ) = K ( K ( x i | z 4 i +1 ) | x i , z 4 i +1 ) > + log m − log log m. (12) First using Lemma 4.1, then applying subsequen tly equations (10) , (11), (12), and (8) gives : K ( u , x, y , v ) 6 X i K ( u i , x i , y i | z 4 i ) + X i K ( v i | z 4 i +3 ) + O ( n ) 6 X i K ( u i , x i | z 4 i ) + X i K ( y i , v i | z 4 i +2 ) − X i K ( y i | z 4 i +2 ) + O ( n log lo g m ) 6 K ( T ux ) + K ( T y v ) − O ( n log m ) . 8 Let h ., . i b e a computable bijectiv e pairing function suc h that for all a, b ∈ ω , l ( h a, b i ) 6 l ( a ) + l ( b ). Let: x ′ i = h u i , x i i y ′ i = h y i , v i i . T o finish the pro of it suffices to sho w that l ( x ′ ) + l ( y ′ ) 6 N 6 O ( nm ) . (13) Remark that b ecause x i ∈ 2 m , l ( x i ) 6 2 m and b ecause y i = K ( T x,i ), l ( y i ) 6 + 2 log m : l ( x ′ i ) + l ( y ′ i ) 6 l ( u i ) + l ( x i ) + l ( y i ) + l ( v i ) 6 l ( D ux,i ) + 2 m + 2 log m + l ( D y v,i ) . Cho ose N = 3 mn . F or X = ux, y v , P i D X,i 6 N + 1, a nd therefore P i l ( D X,i ) 6 3 n log m . This sho ws that for m large enough: l ( x ′ ) + l ( y ′ ) 6 3 mn = N . (14) This show s equation (13). Corollary 4.5. F or some c > 0 , for al l but finitely many n , ther e exist a z ∈ 2 2 n such that: K (0 → z 1 ; ... ; z 2 n − 2 → z 2 n − 1 ) + K ( z 1 → z 2 ; ... ; z 2 n − 1 → z 2 n ) − K ( z ) > cn. (15) Pr o of. Let x ′ , y ′ b e as constructed in t he pro of. Let x ′ i and y ′ i b e binary prefix-free encodings corresp onding to the definition of l ( x ). of x ′ i and y ′ i , i 6 n . Define z : z = x ′ 1 , 1 , 0 , ..., x ′ 1 ,l ( x ′ 1 ) , 0 , 0 , y ′ 1 , 1 , ..., 0 , y ′ 1 ,l ( y ′ 1 ) , ... x ′ n, 1 , 0 , ..., x ′ n,l ( x ′ n ) , 0 , 0 , y ′ n, 1 , ..., 0 , y ′ n,l ( y ′ n ) . Since P i 6 n l ( x ′ i ) + l ( y ′ i ) 6 3 mn , we ha v e that z ∈ 2 6 6 n . This sho ws that for all but finitely man y n a string of length maximally 6 mn exists that satisfies the inequalit y of the lemma. By app ending zeros to the end of x ′ and y ′ , equality (15) can b e satisfied f or ev ery n . Ac knowled gmen t The author is grateful for the commen ts of A. Shen on early pro of attempts and motiv ation to write out a f ull exact pro of. 9