Optimal Succinctness for Range Minimum Queries

Optimal Succinctness for Range Minim um Queries Johannes Fisc her Universit¨ at T¨ ubingen, Center for Bioi nformatics (ZBIT), S and 14, 720 76 T ¨ ubingen fischer@in formatik. uni-tuebin gen.de Abstract. F or a static array A of n totally ordered ob jects, a r ange minimum query asks for t he p osition of the minim um betw een tw o sp ecified array indices. W e sho w how to preprocess A into a sc heme of size 2 n + o ( n ) bits that allo ws to answer range minim u m queries on A in constan t time. This space is asymptotically optimal in the imp ortant setting where access t o A is not p ermitted after the prepro cessing step. O u r sc heme can b e computed in linear time, using only n + o ( n ) additional bits for construction. W e also improv e on LCA- computation in BPS- or DFUDS-en coded trees. 1 In tr o duction F or an array A [1 , n ] of n n atur al n umbers or o ther ob jects from a to tally ordered unive rse, a r ange minimum query rm q A ( i, j ) for i ≤ j return s the p osition of a minim um elemen t in the sub-arra y A [ i, j ]; i.e., rmq A ( i, j ) = argmin i ≤ k ≤ j { A [ k ] } . Th is fundament al algorithmic problem has numerous applications, e.g., in text indexin g [1, 15, 36], text compression [7], d o cument r etriev al [31, 37, 42], flo wgraph s [19], range queries [40] , p osition-restricted pattern matc hing [8], just to mention a few. In al l of these applications, the arra y A in whic h the range min im um queries (RM Qs) are p erformed is static and kno wn in adv ance, which is also the scenario considered in this article. In this case it mak es sense to prepro cess A in to a (pr epro cessing-) scheme su c h that future RMQs can b e answered quic kly . W e can h ence form u late the follo wing problem. Pr oblem 1 (RMQ-Pr oblem). Giv en: a static array A [1 , n ] of n totally ordered ob jects. Compute: an (ideally small) data stru cture, called scheme , that allo ws to answ er RMQs on A in constan t time. The historically first suc h sc heme d ue to Ga b o w et al. [16] is based on the follo w in g idea: b ecause an RMQ-instance can b e transformed into an instance of lowest c ommon anc estors (LCAs) in the Cartesian T r e e [43], one can u se an y linear-time prepro cessing sc heme for O (1)-LCAs [3, 5, 23, 41] in order to answer RMQs in constan t time. The problem of this tr an s formation [16], b oth in t heory a nd in p ractice, can b e see n b y the f ol- lo wing dilemma: storing the Cartesian T ree explicitly (i.e., with lab els and p oin ters) needs O ( n log n ) bits of sp ace, w hile storing it suc cinctly in 2 n + o ( n ) bits [4, 30] do es not allo w to map th e arra y- indices to the corresp onding no des (see Sect. 1.1 for more details on why this is difficult). A su c cinc t data structur e us es space that is close to the information-theoretic low er b ound , in the s ense t hat ob j ects from a univ erse of cardinalit y L are stored in (1 + o (1)) log L bits. 1 Researc h on succinct data s tructures is v ery activ e, and we just men tion some examples from the realm of trees [4, 9, 18, 26, 30, 39], dictionaries [33, 34], and strings [10, 11, 21, 22, 35, 38], b eing well a w are of the fact that this list is far fr om complete. This article presen ts the first succi nct data structure for O (1)-RMQs in the standard w ord-RAM mod el of computation (whic h is also the mod el used in all LCA- and RMQ-sc hemes cited in this article). 1 Throughout this article, space is measured in bits, and log denotes the binary logarithm. T able 1. Prepro cessing sc hemes for O (1)-RMQs, w here | A | den otes the space f or the (read-only) input arra y . reference final space construction space comments [5, 23, 41] O ( n log n ) + | A | O ( n log n ) + | A | ori ginally devised for LCA, but solve RMQ via Cartesian T ree [3] O ( n log n ) + | A | O ( n log n ) + | A | sig nificantly simpler than p revious schemes [2] O ( n log n ) + | A | O ( n log n ) + | A | only solution not based on Cartesian T rees [13] 2 n + o ( n ) + | A | 2 n + o ( n ) + | A | generalizes to 2 c n + o ( n ) + | A | bits, const. c (see F o otn ote 2) [14] O ( nH k ) + o ( n ) 2 n + o ( n ) + | A | H k is the empirical entrop y [28] of A (small if A is compress ible) [36] n + o ( n ) n + o ( n ) only for ± 1 rmq ; A m ust be enco ded as an n -bit- vector [37] 4 n + o ( n ) O ( n log n ) + | A | only non-systematic data structu re so f ar this article 2 n + o ( n ) 3 n + o ( n ) + | A | final space requirement optimal Before d etailing our con tribution, w e fi r st classify and su mmarize existing solutions for O (1)- RMQs. 1.1 Previous Solutions for RMQ In accordance with common nomencla ture [1 7], prepro cessing sc hemes for O (1)-RMQs can b e clas- sified in to t w o different types: systematic and non-systematic . S y s tematic sc hemes m u st store the input arr a y A verbatim along with the additional inform ation for answering the queries. In suc h a case the query algorithm c an consu lt A when answering the queries; th is is in deed wh at all systematic sc h emes mak e h ea vy use of. On the con trary , non-systematic sc hemes must b e able to obtain their fin al answ er without consulting th e arra y . This second type is imp ortan t for at least t wo reasons: 1. In some app lications, e.g., in algorithms for do cumen t retriev al [31, 37] or p osition restricted substring matc hing [8], only the p osition of the minimum matters, b ut not the v alue of this minim um. In s uc h cases it w ould b e a waste of space (b oth in th eory and in practice) to keep the input arra y in memory , just for ob taining the final ans w er to the RMQs, as in the ca se of systematic sc hemes. 2. If the time to access the elemen ts in A is ω (1), this slo wed-do wn access time pr opagates to the time for answ ering RMQs if the query algo rithm consults the input arra y . As a p rominent example, in string pro cessing RMQ is often used in conjunction with the arra y of longest c ommon pr efixes of lexicographically consecutive suffixes, the so-called LCP- arr ay [27]. Ho wev er, storing the LCP-array efficien tly in 2 n + o ( n ) bits [36] increases th e access-ti me to the time needed to retriev e an entry f r om the corresp onding suffix arr ay [27], which is Ω (log ǫ n ) (co nstan t ǫ > 0) at the v ery b est if the suffi x arra y is also s tored in compressed form [21, 35]. Hence, with a systematic scheme the time needed for answering RMQs on LCP could nev er b e O (1) in this case. But exac tly th is w ould b e needed for constan t-time n a vigation in RMQ-based compr essed suffix tr ees [15] (where for differen t reasons th e LCP-arra y is still needed, so this is not th e same as the ab ov e p oin t). In th e follo wing, w e briefly sk etc h previous so lutions for RMQ s c hemes. F or a summary , see Tbl. 1, where, b esides th e fi nal space consumption, in the th ir d column we list the p eak space consump tion at construction time of eac h scheme, whic h sometimes differs from the f orm er term. Systematic Sc hemes. Most sc hemes are based on the Cartesian T ree [43], the only exception b eing th e sc heme due to Alstrup et al. [2]. All direct schemes [2, 3, 13, 36] are b ased on the idea of splitting the qu ery range into seve ral sub-queries, all of wh ic h ha ve b een precomputed, and then returning the o ve rall minim um as the final r esult. T he sc h emes from th e fi rst three ro ws of Tbl. 1 ha ve the same theoretical guaran tees, with Bender et al.’s sc h eme [3] b eing less complex than the previous ones, and Alstru p et al.’s [2] b eing even simpler (and most practica l). The only O ( n )-bit sc heme is d ue to Fisc her and Heun [13] and ac hiev es 2 n + o ( n ) b its of space in addition to the space for the input arra y A . It is based on an “implicit” enumeration of Cartesian T rees only for v ery small blo cks (instead of the wh ole arra y A ). Its further adv ant age is th at it can b e adapted to ac hiev e en tropy-b ounds for compressible inputs [14]. F or systematic schemes, no low er boun d on space is kno wn. 2 An imp ortan t sp ecial case is Sadak ane’s n + o ( n )-bit solution [36] for ± 1 rmq , w h ere it is assumed that A has the prop ert y that A [ i ] − A [ i − 1] = ± 1 for all 1 < i ≤ n , and can hence b e enco ded as a bit-v ector S [1 , n ], where a ‘1’ at p osition i in S indicates that A increases b y 1 at p osition i , and a ‘0’ that it d ecreases. Because w e will make use of this sc h eme in our new algorithm, and also impro v e on its space consumption in S ect. 5, w e will d escrib e it in greater d etail in Sect. 2.2. Non-Systematic Sc hemes. The only existing sc h eme is due to Sadak ane [3 7] and uses 4 n + o ( n ) bits. It is based on the b alanced-paren theses-enco ding (BPS) [30] of th e Cartesian T ree T of the input array A and a o ( n )-LCA-computation therein [36]. The d ifficult y that Sadak ane o vercomes is that in the “o riginal” Cartesian T ree, there is no natural mapp in g b et ween arra y-ind ices in A and p ositions of parentheses (basically b ecause there is n o wa y to d istinguish b et w een left and right no des in the BPS of T ); therefore, S adak an e in tro d uces n “fak e” lea v es to get s uc h a mapp in g. There are tw o main d ra w b ac ks of this solution. 1. Due to the in tro duction of the “fak e” lea ve s , it do es not ac hieve the i nf ormation-the or etic lower b ound (for n on-systematic sc hemes) of 2 n − Θ (log n ) b its. This lo wer b oun d is easy to see b ecause an y scheme for RMQs allo ws to reconstruct the Cartesian T ree by ite rativ ely qu erying the s c heme for the minimum (in analog y to the definition of the Cartesian T ree); and b ecause the Cartesian T ree is binary and e ac h bin ary t ree is a Cartesian T ree for so me input arra y , an y sc heme m ust use at least log(  2 n − 1 n − 1  / (2 n − 1)) = 2 n − Θ (log n ) bits [25]. 2. F or getting an O ( n )-time construction algorithm, the (mo dified) Cartesian T ree n eeds to b e first constru cted in a p oin ter-based implementa tion, and th en con verted to the space-sa ving BPS. This leads to a c onstruction sp ac e r e quir ement of O ( n log n ) bits, as eac h no d e o ccupies O (log n ) bits in memory . The problem why th e BPS cannot b e constructed directly in O ( n ) time (at least w e are not a ware of suc h an algorithm) is th at a “lo cal” c hange in A (b e it only app endin g a new elemen t at the end) do es not n ecessarily lead to a “lo cal” c hange in the tree; this is also the in tuitiv e r eason why main taining dynamic Cartesian T rees is d iffi cu lt [6]. 1.2 O ur Results W e address the t wo aforemen tioned p roblems of Sadak ane’s solution [37] and resolv e them in the follo wing w a y: 2 The claimed lo w er boun d of 2 n + o ( n ) + | A | bits un der the “min-prob e-mo del” [13] turned out to b e wrong, as w as kind ly pointed out to the aut h ors by S. Sriniv asa Rao (personal communicatio n, Nov em b er 2007 ). In fact, it is easy to lo w er the space consumption of [13] to 2 c n + o ( n ) + | A | bits (constant in t eger c > 0) by grouping c adjacent elemen ts in A ’s blocks together, and “building” the Cartesian T rees only on the minima of these groups. 1. W e introd uce a new prepr o cessing scheme for O (1)-RMQs that o ccupies only 2 n + o ( n ) b its in memory , thus b eing the fir st that asymptotically ac hieve s the in formation-theoretic low er b ound for non-systematic sc hemes. The critica l reader migh t call this “lo wering the constan ts” or “mic ro-optimizatio n,” but w e b eliev e that data stru ctures us in g the smallest p ossible space are of high imp ortance, b oth in theory and in practice. And indeed, there are many examples of this in literature: for instance, Mun r o and Raman [30] giv e a 2 n + o ( n )-bit-solution for represent ing ord ered trees, w hile sup p orting most na vigational op erations in constan t time, although a O ( n )-bit-solution (roughly 10 n bits [30]) h ad al ready b een kno wn for some 10 y ears b efore [25]. Another example comes from compr essed text indexing [32], w here a lot of effort has b een p ut into ac hieving indexes of size nH k + o ( n log σ ) [11], although indexes of size O ( nH k ) + o ( n log σ ) had b een kno wn earlier [10, 22, 35]. (Here, H k is the k -th-order empirical en tropy of the input text T [28] and measures the “compressibilit y” of T , wh ile σ is T ’s alphab et size.) 2. W e give a dir e ct construction algorithm for the ab o ve scheme that needs only n + o ( n ) b its of space in addition to the sp ace for the fi nal scheme, thus lo we ring the construction space f or non- systematic sc hemes from O ( n log n ) to O ( n ) b its (on top of A ). This is a s ignifican t impro v ement, as the space for storing A is not necessarily Θ ( n log n ); for example, if the n um b ers in A are in tegers in the range [1 , log O (1) n ], A can b e stored as an arr a y of pac ked w ords using only O ( n log log n ) bits of space. S ee Sect. 6 for a differen t example. T he construction sp ace is an imp ortant iss ue and often limits the pr acticali t y of a data structure, esp ecially for large inp uts (as they arise no w ada ys in w eb-page-analysis or computational biology). The intuitiv e explanation why our sc h eme works b etter than S adak ane’s s c heme [37] is that ours is based on a new tree in whic h the preorder-n u m b ers of the no des corresp ond to t he arra y-indices in A , thereb y rendering the in tro duction of “fak e” lea v es (as describ ed earlier) un necessary . In summary , this article is dev oted to p ro ving Theorem 1. F or an arr ay A of n obje cts fr om a total ly or der e d universe, ther e is a pr epr o c essing scheme for O (1) -RMQ s on A that o c cupies only 2 n + O ( n log log n log n ) bi ts of memo ry, while not ne e ding ac c ess to A after its c onstruction, thus me eting the information-the or etic lower b ound. This scheme c an b e c onstructe d in O ( n ) time, using only n + o ( n ) bits of sp ac e in addition to the sp ac e for the input and the final scheme. This r esult is not only app ealing in theory , but also imp ortan t in practice . F o r example, wh en RMQs are used in conjunction with sequences of DNA (g enomic data), where the alphab et size σ is 4, storing the DNA ev en in unc ompr esse d form tak es only 2 n bits, already less than th e 4 n b its of Sadak ane’s solution [37]. Hence, halving the space for RMQ s leads to a s ignifi can t redu ction of total space. F urther, b ecause n is t ypically v ery large ( n ≈ 2 32 for the human genome), a construction space of O ( n log n ) bits is m uc h higher than th e O ( n log σ ) bits for the DNA itself. An additional (practical) adv anta ge of our new sc heme is that it also h alv es the sp ace of th e lo wer order terms (“ o (2 n ) vs. o (4 n ) bits”). T h is is particularly relev an t for realistic p roblem sizes, where the lo wer order terms dominate the linear term. An implementati on in C++ of our new scheme can b e d o wn- loaded from h ttp://ww w- ab.informat ik.uni- tuebingen.de/people/fischer/optimalRMQ.tgz . 1.3 O utline Sect. 2 presents some basic to ols. Sect. 3 in tro duces the n ew prepro cessing scheme. Sect. 4 addresses the linear-time construction of the s c heme. Sect. 5 lo wers the second-order term b y giving a new data stru cture for LCA-computation in succinct tr ees. Sect. 6 sho w s a concrete example of an application wh er e our new p repro cessing scheme impro ves on the total space. 2 Preliminaries This section sket c hes some known data structur es that we are going to make use of. Through ou t this article, w e use the standard w or d-RAM m o del of computation, where fundamen tal arithmetic op erations on words consisting of Θ (log n ) consecutiv e b its can b e compu ted in O (1) time. 2.1 Rank and Select on Binary Strings Consider a bit-string S [1 , n ] of length n . W e d efine th e fund amen tal r ank - and sele ct -op erations on S as follo ws: r ank 1 ( S, i ) giv es the n u m b er of 1’s in the prefix S [1 , i ], and sele ct 1 ( S, i ) giv es the p osition of the i ’th 1 in S , reading S fr om left to righ t (1 ≤ i ≤ n ). Op erations r ank 0 ( S, i ) and sele ct 0 ( S, i ) are defin ed similarly f or 0-bits. There are data str uctures of size O ( n log log n log n ) bits in addition to S that su pp ort rank- and select-op erations in O (1) time [29]. 2.2 Dat a Structures for ± 1RMQ Consider an arra y E [1 , n ] of natur al num b ers , wh ere the d ifference b et we en consecutiv e elemen ts in E is either +1 or − 1 (i.e. E [ i ] − E [ i − 1] = ± 1 for all 1 < i ≤ n ). Such an array E can b e enco ded as a bit-v ector S [1 , n ], w h ere S [1] = 0, and for i > 1, S [ i ] = 1 iff E [ i ] − E [ i − 1] = + 1. Then E [ i ] can b e obtained by E [1] + r ank 1 ( S, i ) − r ank 0 ( S, i ) + 1 = E [1] + i − 2 r ank 0 ( S, i ) + 1. Under this setting, Sadak ane [36] sh ows h o w to supp ort RMQs on E in O (1) time, us in g S and additional structures of size O ( n log 2 log n log n ) bits. W e will improv e this space to O ( n log log n log n ) in Sect. 5. A tec hnical detail is that ± 1 rmq ( i, j ) yields the p osition of the leftmost minimum in E [ i, j ] if there are multiple occurr ences of this m inim u m. 2.3 Sequences of Balanced P arentheses A string B [1 , 2 n ] of n op enin g paren th eses ‘(’ and n closing paren th eses ‘)’ is called b alanc e d if in eac h prefix B [1 , i ], 1 ≤ i ≤ 2 n , the num b er of ‘)’s is no more than the num b er of ‘(’s. Op er- ation findop en ( B , i ) return s the p osition j of the “matc hing” op ening parenthesis for the clos- ing p aren thesis at p osition i in B . This p osition j is defi ned as th e largest j < i for w hic h r ank ( ( B , i ) − ran k ) ( B , i ) = r ank ( ( B , j ) − r ank ) ( B , j ). The findop en -o p eration can b e computed in constan t time [30]; the most s pace-efficien t data structure for this n eeds O ( n log log n log n ) bits [18]. 2.4 Dept h-First Unary Degree Enco ding of Ordered T rees The Depth-First Unary Degree Sequence (DFUDS) U of an ordered tree T is defined as follo w s [4]. If T is a leaf, U is g iv en by ‘() ’. Otherwise, if the ro ot of T has w subtrees T 1 , . . . , T w in this ord er, U is giv en by the juxtap osition of w + 1 ‘(’s, a ‘)’, and the DFUDS’s of T 1 , . . . , T w in th is order, with the first ‘(’ of eac h T i b eing omitted. It is easy to see that the resulting sequence is balanced, and that it can b e in terp r eted as a preorder-listing of T ’s n o des, w h ere, ignoring the v er y first ‘(’, a no de with w c h ildren is encoded in unary as ‘( w )’ (hence the name DFUD S). 3 The N ew Pr epro cessing Sc heme W e are now ready to dive into the tec hnical details of our n ew prepr o cessing sc heme. The basis will b e a new tree, the 2d-Min-H e ap , defin ed as f ollo ws. Recall that A [1 , n ] is the arra y to b e prepro cessed for RMQs. F or tec h n ical reasons, we d efine A [0] = −∞ as th e “artificial” ov erall minim um. ( (( ( ( ) 4 5 4 3 2 1 14 15 16 13 12 11 10 9 7 8 4 5 6 1 2 3 0 ( 3 2 34 3 2 34 2 ) 1 3 ( 1 2 ) 1 ( ) 2 3 ) ( 3 2 ) 2 ) ( 4 3 ) 3 1 3 2 4 2 5 3 5 5 4 5 4 3 4 3 5 4 6 7 0 1 2 3 5 8 9 10 11 12 13 14 16 A U E = = = 2 2 ) ) ) ) ) ) 0 1 ( (( ) ( (( ) 15 i x y j i x y ( ) 3 2 j 0 0 − Fig. 1. T op: The 2d-Min-Heap M A of the in put array A . Bottom: M A ’s DFUDS U and U ’s excess sequence E . Two example queries rmq A ( i, j ) are und er lin ed, includin g their corresp ondin g queries ± 1 rmq E ( x, y ). Definition 1. The 2d-Min-Heap M A of A is a lab ele d and or der e d tr e e with vertic es v 0 , . . . , v n , wher e v i is lab ele d with i for al l 0 ≤ i ≤ n . F or 1 ≤ i ≤ n , the p ar ent no de of v i is v j iff j < i , A [ j ] < A [ i ] , and A [ k ] ≥ A [ i ] for al l j < k ≤ i . The or der of the childr en is chosen suc h that their lab els ar e incr e asing fr om left to right. Observe th at this is a w ell-defined tree with the ro ot being alwa ys lab eled as 0 , and th at a no de v i can b e uniquely iden tified by it s label i , wh ich we will do hen ceforth. See Fig . 1 for an example. W e note the follo wing useful prop er ties of M A . Lemma 1. L et M A b e the 2d -Min-H e ap of A . 1. The no de lab els c orr esp ond to the pr e or der-numb ers of M A (c ounting start s at 0). 2. L et i b e a no de in M A with childr en x 1 , . . . , x k . Then A [ i ] < A [ x j ] for al l 1 ≤ j ≤ k . 3. A gain, let i b e a no de in M A with childr en x 1 , . . . , x k . Then A [ x j ] ≤ A [ x j − 1 ] f or al l 1 < j ≤ k . Pr o of. Because t he root of M A is a lw ays lab eled with 0 and the order of th e c hildren is ind uced by their lab els, prop ert y 1 holds. Prop ert y 2 follo ws imm ediately from Def. 1. F or prop ert y 3, assume for the sak e of con tradiction that A [ x j ] > A [ x j − 1 ] for t wo c hildren x j and x j − 1 of i . F rom prop ert y 1, we kno w that i < x j − 1 < x j , contradicting the definition of the paren t-c h ild-relationship in M A , whic h sa ys that A [ k ] ≥ A [ x j ] for all i < k ≤ x j . ⊓ ⊔ Prop erties 2 and 3 of the ab ov e lemma explain the choic e of the name “2d-Min-Heap,” b ecause M A exhibits a m inim u m-prop erty on b oth the parent-c h ild- and the sibling-sibling-relationship, i.e., in tw o dimensions. The follo wing lemma will b e cen tral for our sc h eme, as it giv es the desired connection of 2d - Min-Heaps and RMQs. Lemma 2. L et M A b e the 2d-Min-H e ap of A . F or arbitr ary no des i and j , 1 ≤ i < j ≤ n , let ℓ denote the LCA of i and j in M A (r e c al l that we identify no des with their lab els). Then if ℓ = i , rmq A ( i, j ) is giv e n by i , and otherwise, r mq A ( i, j ) is given by the child of ℓ that is on the p ath fr om ℓ to j . Pr o of. F or an arbitrary no d e x in M A , let T x denote the subtree of M A that is r o oted at x . Ther e are t wo cases to p r o ve. ℓ = i. T his means that j is a descendan t of i . Due to prop er ty 1 of Lemma 1, this implies that all no des i, i + 1 , . . . , j are in T i , and the recursiv e application of prop ert y 2 implies t hat A [ i ] is the minim um in the query range [ i, j ]. ℓ 6 = i. L et x 1 , . . . , x k b e the c h ildren of ℓ . F urth er, let α and β (1 ≤ α ≤ β ≤ k ) b e defined such that T x α con tains i , and T x β con tains j . Because ℓ 6 = i and pr op ert y 1 of L emma 1, w e must ha ve ℓ < i ; in other words, the LCA is not in the query range. But also du e to prop ert y 1, eve ry no de in [ i, j ] is in T x γ for some α ≤ γ ≤ β , and in particular x γ ∈ [ i, j ] for all α < γ ≤ β . T aking this together with prop ert y 2, w e see that { x γ : α < γ ≤ β } are th e only cand id ate p ositions for the minim um in A [ i, j ]. Due to prop erty 3, w e see that x β (the c h ild of ℓ on the path to j ) is the p osition where the o verall minimum in A [ i, j ] o ccurs. ⊓ ⊔ Note that (unlike for ± 1 rmq ) th is algorithm yields the rightmost minim um in th e query range if this is not unique. Ho wev er, it can b e e asily arranged to return the leftmost minimum b y adapting the defin ition of the 2d-Min-Heap, if this is desired. T o ac hieve the optimal 2 n + o ( n ) bits for our sc heme, we represent the 2d-Min-Heap M A b y its DFUDS U and o ( n ) structures for r ank ) -, sele ct ) -, and findop en -op erations on U (see Sect. 2). W e fu rther n eed structures for ± 1 rmq on the exc ess- se quenc e E [1 , 2 n ] of U , defined as E [ i ] = r ank ( ( U, i ) − r ank ) ( U, i ). Th is s equ ence clearly satisfies the prop ert y that sub sequen t element s differ b y exactly 1, and is already enco d ed in the righ t form (b y means of the DFUDS U ) for applying the ± 1 rmq -sc h eme fr om Sect. 2.2. The reasons for p referring the DFUDS ov er the BPS-repr esen tation [30] of M A are (1) the op erations needed to p erf orm on M A are particularly easy on DFUDS (see the next corollary), and (2) we h a ve found a f ast and space-efficien t algorithm for constru cting the DFUDS directly (see the next section). Corollary 1. Given the DFUDS U of M A , rmq A ( i, j ) c an b e answer e d in O (1) time by the fol- lowing se quenc e of op er ations ( 1 ≤ i < j ≤ n ). 1. x ← sele ct ) ( U, i + 1) 2. y ← sele ct ) ( U, j ) 3. w ← ± 1 rmq E ( x, y ) 4. if r ank ) ( U, findop en ( U, w )) = i then return i 5. else return r ank ) ( U, w ) Pr o of. Let ℓ b e the true L C A of i and j in M A . I n sp ecting the details of ho w LCA-c omputation in DFUDS is d one [26, Lemma 3.2], we see that after the ± 1 rmq -call in line 3 of the ab o v e algorithm, w + 1 con tains the starting p osition in U of the enco d ing o f ℓ ’s c hild that is on the path to j . 3 Line 4 c hec ks if ℓ = i b y co mparing their preorder-n u m b ers and returns i in that case (ca se 1 of Lemma 2) — it follo ws from the description of the p aren t-op eration in the original article on DFUDS [4] that this is correct. Finally , in line 5, the p r eorder-n um b er of ℓ ’s c hild that is on th e path to j is computed correctly (case 2 of Lemma 2). ⊓ ⊔ W e h a ve sh o wn these op erations so explicitly in ord er to emphasize the simplicit y of our ap- proac h. Note in particular that not all op erations on DFUDS ha ve to b e “implement ed” for our RMQ-sc heme, and th at we find the correct child of the LCA ℓ directly , without fin ding ℓ explicitl y . W e encourage the r eader to w ork on the examples in Fig. 1, where th e r esp ectiv e RMQs in b oth A and E are und erlined and lab eled with the v ariables fr om Cor. 1. 3 In line 1, we correct a minor error in th e original article [26] by computing the starting p osition x slightly d ifferently , whic h is necessary in th e case t h at i = lca ( i, j ) (confirmed by K. Sadak ane, p ersonal communicatio n, Ma y 2008). 4 Construction of 2d-Min-Heaps W e no w show ho w to construct the DFUDS U of M A in linear time and n + o ( n ) bits of extra space. W e first giv e a general O ( n )-time algorithm that uses O ( n log n ) bits (Sect. 4.1), and then sho w ho w to reduce its space to n + o ( n ) bits, while still having linear runn ing time (Sect. 4.2). 4.1 T he General Linear-Time Algorithm W e show how to construct U (the DFUD S of M A ) in linear time. The idea is to scan A from right to left and build U fr om righ t to left, to o. Su p p ose w e are currently in step i ( n ≥ i ≥ 0), and A [ i + 1 , n ] h av e already b een scanned. W e keep a stac k S [1 , h ] (wh ere S [ h ] is the top) with the prop erties th at A [ S [ h ]] ≥ · · · ≥ A [ S [1]], and i < S [ h ] < · · · < S [1] ≤ n . S con tains exactly those indices j ∈ [ i + 1 , n ] for w h ic h A [ k ] ≥ A [ j ] for all i < k < j . Initially , b oth S and U are empty . When in step i , w e fi rst write a ‘)’ to the current b eginning of U , and then p op all w in dices from S for which the corresp onding en try in A is strictly greater than A [ i ]. T o reflect this c hange in U , w e write w op ening parentheses ‘(’ to the current b eginning of U . Finally , w e pu sh i on S and m o ve to the next (i.e. pr eceding) p osition i − 1. It is easy to see th at these c hanges on S m ain tain the prop erties of the stac k. If i = 0, w e write an initial ‘(’ to U and stop the alg orithm. The correctness of this algorithm follo ws f rom the fact th at due to the d efinition of M A , the degree of no de i is gi v en by the n u m b er w of arra y-indices to the right of i which ha ve A [ i ] as their closest sm aller v alue (prop erties 2 and 3 of Lemma 1). Th us, in U no d e i is enco d ed as ‘( w )’, wh ic h is exactly wh at w e do. Because eac h ind ex is pus h ed and p opp ed exactly once on/from S , the linear runn in g time follo ws. 4.2 O ( n )-bit Solut ion The only drawbac k of the ab o v e algorithm is that stac k S requires O ( n log n ) bits in th e w orst case. W e solv e th is problem b y represent ing S as a bit-ve ctor S ′ [1 , n ]. S ′ [ i ] is 1 if i is on S , and 0 otherwise. In order to m aintain constant time access to S , we use a standard blo c king-tec hniqu e as follo ws. W e logically group s = ⌈ log n 2 ⌉ consecutiv e elemen ts of S ′ in to blo cks B 0 , . . . , B ⌊ n − 1 s ⌋ . F urth er, s ′ = s 2 elemen ts are group ed into sup er-blo cks B ′ 0 , . . . , B ′ ⌊ n − 1 s ′ ⌋ . F or eac h suc h (sup er-)blo ck B that cont ains at least on e 1, in a n ew table M (or M ′ , resp ectiv ely) at p osition x we store the blo c k num b er of the leftmost (sup er-)blo ck to the righ t of B that conta ins a 1, in M only relativ e to the b eginning of the sup er -b lo c k. Th ese tables need O ( n s log( s ′ /s )) = O ( n log log n log n ) and O ( n s ′ log( n /s )) = O ( n log n ) bits of space, resp ectiv ely . F u r ther, for all p ossible bit- v ectors of length s we main tain a table P that stores the p osition of the leftmost 1 in that v ector. This table needs O (2 s · log s ) = O ( √ n log log n ) = o ( n ) bits. Next, we sh o w ho w to use these tables for constan t-time access to S , and ho w to ke ep M and M ′ up to d ate. When en tering step i of the algorithm, we kno wn that S ′ [ i + 1] = 1, b ecause p osition i + 1 has b een pushed on S as the last op eration of the previous step. Thus, the top of S is giv en by i + 1. F or finding the leftmost 1 in S ′ to the r igh t of j > i (p osition j has just b een p opp ed from S ), w e first c heck if j ’s blo c k B x , x = ⌊ j − 1 s ⌋ , con tains a 1, and if so, fin d this leftmost 1 b y consu lting P . If B x do es not con tain a 1, we jump to the n ext blo ck B y con taining a 1 b y fi rst j umping to y = x + M [ x ], a nd if this blo c k does n ot con tain a 1, by further ju m ping to y = M ′ [ ⌊ j − 1 s ′ ⌋ ]. In blo ck y , we can again use P to fi nd the l eftmost 1. Thus, we ca n find the new top of S in constan t time. In order to keep M up to date, we need to handle the op erations where (1) e lemen ts are pushed on S (i.e., a 0 is changed to a 1 in S ′ ), and (2) elements are p opp ed from S (a 1 changed to a 0). Because in step i only i is pushed on S , for op eration (1) we jus t n eed to store the b lo c k num b er y of the former top in M [ x ] ( x = ⌊ i − 1 s ⌋ ), if this is in a different b lo c k (i.e., if x 6 = y ). Changes to M ′ are similar. F or op eration (2), nothing h as to b e done at all, b ecause ev en if the p opp ed index w as the last 1 in its (sup er-)block, we kno w that all (sup er-)blo cks to the left of it do not con tain a 1, so no v alues in M and M ′ ha ve to b e c hanged. Not e that this only wo rks b ecause elemen ts to the righ t of i will nev er b e pushed ag ain onto S . This completes the d escription of the n + o ( n )-bit construction algorithm. 5 Lo wering the Second-Order-T erm Un til now, the second-order-term is dominated by the O ( n log 2 log n log n ) bits from S adak ane’s prep r o- cessing sc h eme for ± 1 rmq (Sect. 2.2), wh ile all other terms (for r ank , sele ct and findop en ) are O ( n log log n log n ). W e sho w in this section a simple wa y to lo wer the space for ± 1 rmq to O ( n log log n log n ), thereb y completing the pro of o f Th m. 1. As in the original algorithm [36], w e divide the in put arra y E int o n ′ = ⌊ n − 1 s ⌋ blo c ks of size s = ⌈ log n 2 ⌉ . Queries are deco m p osed in to at most thr ee non-o verlapping su b-queries, where the first and the last sub -queries are inside of the b lo c ks of size s , and the midd le on e exactly spans o ve r blo c ks. The tw o qu eries i nside of the blo cks are answered by table lookup s u s ing O ( √ n log 2 n ) bits, as in the original alg orithm. F or th e queries spanning exactly o v er blo c ks of size s , we pro ceed as follo ws. Define a new arra y E ′ [0 , n ′ ] suc h that E ′ [ i ] holds the minimum of E ’s i ’th block. E ′ is repr esen ted only impl icitly b y an arra y E ′′ [0 , n ′ ], where E ′′ [ i ] holds the p osition of the m in im u m in the i ’th blo ck, relativ e to the b eginnin g of that b lo c k. T hen E ′ [ i ] = E [ is + E ′′ [ i ]]. Because E ′′ stores n/ log n n u m b ers from the range [1 , s ], the size for storing E ′ is th us O ( n log log n log n ) bits. Note that u nlik e E , E ′ do es not necessarily fulfill the ± 1-prop erty . E ′ is no w prep ro cessed for constan t-time RMQs with the systematic sc heme of Fisc h er and Heun [13], using 2 n ′ + o ( n ′ ) = O ( n log n ) bits of space. Thus, b y querying r mq E ′ ( i, j ) for 1 ≤ i ≤ j ≤ n ′ , we can also find the minima for the sub-queries sp anning exactly o ver the blo cks in E . Tw o commen ts a re in ord er at this p lace. First, the used RMQ -scheme [13] do es allo w the in put arra y to b e represen ted implicitly , as in our case. And second, it do es not use S adak ane’s solution for ± 1 rmq , so ther e are no circular dep end encies. As a corollary , th is approac h also lo wers the space for LCA-computation in BPS [36] and DFUDS [26] from O ( n log 2 log n log n ) to O ( n log log n log n ), as these are b ased on ± 1 rmq : Corollary 2. Given the BPS or DFUDS of an or der e d tr e e T , ther e is a data structur e of size O ( n log log n log n ) bits that al lows to answer LCA-queries i n T in c onstant time. 6 Application in Do cumen t Retriev al Systems W e no w sk etc h a concrete example of where Th m. 1 lo wers the construction space of a d ifferen t data stru cture. This section is mean t to sho w that there a re ind eed ap p lications where the memory b ottlenec k is the construction sp ace for RMQs. W e consider the follo w ing p roblem: Pr oblem 2 (Do cument Listing Pr oblem [31]). Giv en: a collectio n of k text do cuments D = { D 1 , . . . , D k } of total length n . Compute: an index that, giv en a searc h pattern P of length m , returns all d d o cument s from D that con tain P , in time p rop ortional to m and d (in con trast to al l o ccurr en ces of P in D ). Sadak ane [37, Sect. 4] giv es a succinct index for this p roblem. It uses three p arts, for conv enience listed here together with their final size: – Compressed suffi x arra y [35] A of the concatenatio n of all k do cuments, | A | = 1 ǫ H 0 n + O ( n ) bits. – Arra y of do cument iden tifiers D , defined b y D [ i ] = j iff the A [ i ]’th suffix “b elongs to” do cument j . Its size is O ( k log n k ) bits – Range minim um queries on an array C , | RM Q | = 4 n + o ( n ) bits. Here, C stores p ositions in A of nearest previous occurren ces of indexed p ositions from the same do cument, C [ i ] = max { j < i : D [ j ] = D [ i ] } . In t he query algorithm, only the p ositions of the minima matter; h ence, this is a non-systematic setting. Apart from halving the sp ace f or RMQ from 4 n to 2 n bits, our n ew scheme also low ers the p eak space consump tion of Sadak ane’s ind ex for the Do cumen t Listing Prob lem. Let us consid er the c onstruction time and space for eac h part in tu r n: – Arra y A can b e bu ilt in O ( n ) time and O ( n ) bits (constan t alphab et), or O ( n log log | Σ | ) time using O ( n log | Σ | ) b its (arbitrary alphab et Σ ) of space [24]. – Arra y D is actually imp lemen ted as a fully ind exable dictionary [34] called D ′ , and can certainly b e built in linear time using O ( n ) b its wo r king space, as we can alwa y s couple the blo c k - enco dings [34] with the o ( n )-bit structures for un compr essed solutions for rank and select [29]. – As already men tioned b efore, for a fast construction of Sadak ane’s sc heme for O (1)-RMQs on C , we wo uld ha ve needed Θ ( n log n ) bits. Ou r new method lo wers this to O ( n ) bits constru ction space. Note that arra y C needs never b e stored plainly durin g the construction: b ecause C is scanned only once when bu ild ing th e DFUDS (Sect. 4) and is th u s acc essed only sequent ially , w e only need to store th e p ositions in A of the last seen d o cument identi fier for eac h of the k do cuments. This can b e done us in g a plain arra y , so | C | = O ( k log n ) bits. In summ ary , we get: Theorem 2. The c onstruction sp ac e for Sadakane’s Index for Do cu ment Listing [37] is lower e d fr om O ( n log n ) bits to O ( n + k log n ) bits (c onstant al phab et) or O ( n log | Σ | + k log n ) bits (a rbitr ary alphab et Σ ) with our scheme for RMQs fr om Thm. 1, while not incr e asing the c onstruction time. This is esp ecial ly interesting if k , the n umb er of do cuments, is not too large , k = O ( n log | Σ | log n ). 7 Concluding Remarks W e ha v e giv en the fir s t optimal prepro cessing scheme for O (1)-RMQs under the imp ortant as- sumption that the inp ut arr a y is n ot av ailable after prepro cessing. T o the exp ert, it might come as a surpr ise that our algorithm is not based on the Cartesian T ree, a concept that has p ro ved to b e v ery successful in former schemes. In stead, w e h a ve in tr o duced a n ew tree, the 2d-Min-Heap, whic h seems to b e b etter suited for our task. 4 W e hop e to ha v e thereb y introdu ced a new v ersatile data stru cture to the algorithms comm unity . And indeed, w e are already a w are of the fact that the 4 The Cartesian T ree and the 2d-Min-Heap are certainly related, as they are b oth obtained from the arra y , and it w ou ld certa inly be p ossible to derive the 2d-Min-H eap (or a related structure obtained from the n atu ral bijection b etw een binary and ordered ro oted trees) fro m the Cartesian T ree, and then conv ert it to the BPS/DFUDS. But see the second point in Sect. 1.2 why th is is not a go o d ide a. 2d-Min-Heap, made public via a prepr in t of this article [12], is piv otal to a new data stru ctur e for succinct trees [39]. W e lea v e it as an op en r esearc h problem whether the 3 n + o ( n )-bit construction s p ace b e lo w ered to an optimal 2 n + o ( n )-bit “in-place” construction algorithm. (A simple example shows that it is not p ossible to use the leading n bits of the DFUDS for the stac k.) 8 Recen t Dev elopmen ts It h as rec en tly b een sho wn [20] that replacing the ± 1 rmq -call in Cor. 1 by the r ange r estricte d en- close -op er ation is adv an tegeous in practice, a s this lat ter op eration can be implemented by sharing the most consuming p arts of the data structur es with those of the findop en-op eration. Ac kno wledgmen t s The author wish es to thank V olk er Heun, V eli M¨ akinen, Gonzalo Na v arro, and the anon ymous LA TIN’10-referees for their helpful commen ts on th is article. References 1. M. I. Ab ouelho da, S. K urtz, and E. Ohlebusc h . Replacing suffix trees with enhanced suffix arra ys. J. Discr ete Algor ithms , 2(1):53–8 6, 20 04. 2. S. Alstrup , C. Gav oille, H. Kaplan, and T. Rauhe. Nearest common ancestors: A surv ey a nd a n ew distributed algorithm. In Pr o c. SP AA , pages 258–264. ACM Press, 2002. 3. M. A. Bend er, M. F arac h -Colton, G. Pemmasa ni, S. Skiena, and P . Sumazin. Lo west common ancestors in trees and directed acyclic graphs. J. A lgorithms , 57(2):75–94, 2005. 4. D. Benoit, E. D. Demaine, J. I. Munro, R. Raman, V. Raman, and S. S. Rao. Representing trees of higher d egree. Algor ithmic a , 43(4):275–29 2, 2005. 5. O. Berkman and U. Vishkin. Recursive star-tree parallel data structure. SIAM J. Comput. , 22(2):221– 242, 1993. 6. I. Bialynic k a-Birula and R . Grossi. Amortized rigidness in dyn amic Cartesian Trees. I n Pr o c. ST ACS , v olume 3884 of LNC S , pages 80–91. Springer, 2006. 7. G. Chen, S. J. Puglisi, and W. F. S myth. LZ factorization using less time and sp ace. M athematics in Comput er Scienc e , 1(4):605–623 , 200 7. 8. M. Cro chemore, C. S. I liop oulos, M. Ku bica, M. S. Rahman, and T . W alen. Improv ed algorithms for t he range next val ue problem and applications. I n Pr o c. ST ACS , pages 205–216. IBFI Schloss Dagstuhl, 2008. 9. P . F erragina, F. Luccio, G. Manzini, and S. Mut hukrishnan. St ructuring lab eled trees for optimal succinctn ess, and beyond. In Pr o c. F OCS , pages 184–196. IEEE C omputer Society , 2005. 10. P . F erragina and G. Manzini. Indexing compressed text. J. A CM , 52(4):552–581, 20 05. 11. P . F erragina, G. Manzini, V. M¨ akinen, and G. Na v arro. Compressed represen tations of sequences and full-text indexes. ACM T r ansactions on Algorithms , 3(2):Article No. 20, 2007. 12. J. Fisc h er. Opt imal su ccinct n ess for range minim u m queries. CoRR, abs/0812.277 5, 2008. 13. J. Fischer and V. Heun. A new succinct representation of RMQ-information and impro vements in the enhanced suffix arra y . In Pr o c. ESCAPE , vo lume 4614 of LNCS , pages 459–470. Springer, 2007. 14. J. Fischer, V. Heun, and H. M. St ¨ uhler. Practical entrop y b ou n ded sc h emes for O (1)-range minimum queries. In Pr o c. DCC , pages 272–281. IEEE Press, 2008. 15. J. Fisc her, V. M¨ akinen, and G. Na va rro. An(other) entrop y- b ounded compressed suffix tree. In Pr o c. CPM , vol ume 5029 of LNCS , pages 152–165. Springer, 2008. 16. H. N. Gab ow, J. L. Bentley , and R. E. T arjan. Scaling and rela ted tec hniqu es for geometry problems. I n Pr o c. STOC , p ages 135–143. AC M Press, 1984. 17. A. G´ al and P . B. Miltersen. The cell prob e complexity of succinct data structures. The or. Comput. Sci. , 379(3):405 –417, 20 07. 18. R. F. Geary , N. Rahman, R. Raman, and V. Raman. A simple op t imal representation for balanced paren theses. The or. Comput. Sci. , 368(3):231– 246, 200 6. 19. L. Georgiadis and R. E. T arjan. Find ing dominators revisited: extended abstract. In Pr o c. SODA , p ages 869–878. ACM /SIAM, 2004. 20. S. Gog and J. Fischer. Ad van tages of shared data structures for sequ ences of balanced parentheses. Sub m itt ed , 2009. 21. R. Grossi, A. Gu pta, and J. S. Vitter. High-order entrop y-compressed text indexes. I n Pr o c. SODA , pages 841–850 . ACM/SIAM, 2003. 22. R. Gross i and J. S. Vitter. C ompressed suffix arrays and suffix trees with applications to tex t indexing and string matc hing. SIAM J. Comput. , 35(2):378–40 7, 2005. 23. D. Harel and R. E. T arjan. F ast algorithms for find ing nearest common ancestors. SIAM J. Comput. , 13(2):3 38– 355, 1984. 24. W.-K. Hon, K. Sadak ane, and W.-K. Su ng. Breaking a time-and-space barrier in constructing full-text indices. SIAM J. Comput. , 38(6):2162–2178, 2009 . 25. G. Jacobson. Space-efficient static trees and graphs. I n Pr o c. F O CS , pages 549– 554. IEEE Computer Society , 1989. 26. J. Jansson, K . Sadak ane, and W.-K. Sung. U ltra-succinct representatio n of ordered trees. In Pr o c. SODA , pages 575–584 . ACM/SIAM, 2007. 27. U. Manber and E. W. Myers. Suffix arra ys: A new metho d for on-line string searc hes. SIAM J. Comput. , 22(5):935– 948, 199 3. 28. G. Manzini. An analysis of the Burrow s-Wheeler transform. J. ACM , 48(3):407– 430, 200 1. 29. J. I. Munro. T ables. In Pr o c. FSTTCS , volume 1180 of LNCS , pages 37–42 . S pringer, 1996. 30. J. I. Mu n ro and V. Raman. Su ccinct representatio n of balanced parentheses and static trees. SIAM J. Comput. , 31(3):762– 776, 200 1. 31. S. Muthukrishnan. Efficien t algorithms for docu ment retriev al problems. In Pr o c. SODA , pages 657–666 . ACM /SIAM, 2002. 32. G. Nav arro and V. M¨ akinen. Compress ed full-text ind exes. ACM Computing Surveys , 39(1):Article No. 2, 2007. 33. R. P agh. Low redund ancy in static dictionaries with constant q uery time. SIAM J. Comput. , 31(2):353–363, 2001. 34. R. Raman, V. Raman, and S. S. R ao. Succinct ind exable dictionaries with applica tions to encoding k - ary trees and m ultisets. ACM T r ansactions on A lgorithms , 3(4):Article No. 43, 2007. 35. K. Sadak ane. New text indexing functionalities of the compressed suffix a rra ys. J. Algorithms , 48(2):294 –313, 2003. 36. K. Sadak ane. Compressed suffix trees with full functionalit y . The ory of Computing Systems , 41(4):5 89–607, 2007. 37. K. Sadak ane. Succinct data stru ct ures for fl ex ible tex t retriev al systems. J. Discr ete A lgorithms , 5(1):12– 22, 2007. 38. K. Sadak ane and R . Grossi . Squeezing succinct data structures in to entrop y b ounds. In Pr o c. SOD A , pages 1230–12 39. A CM/SIAM, 2006. 39. K. Sadak ane and G. N a v arro. F ully- functional succinct trees. Accepted for SODA’10. See also CoRR, abs/0905. 0768v1, 200 9. 40. S. Saxena. Dominance made simple. I nform. Pr o c ess. L ett. , 109(9):409–42 1, 2009. 41. B. Sc hieb er and U. Vishkin. On finding lo w est common ancestors : Simplification and parallelization. SIAM J. Comput. , 17(6):1253 –1262, 1 988. 42. N. V¨ alim¨ aki a nd V. M¨ akinen. Sp ace-efficien t algorithms for do cument retriev al. In Pr o c. CPM , vol ume 4580 of LNCS , pages 205–2 15. Springer, 2007. 43. J. V u illemin. A unifying look at data structures. Comm. ACM , 23(4):229– 239, 198 0.