Optimally Efficient Prefix Search and Multicast in Structured P2P Networks

T ec hnical Rep ort TUD-CS-2008-103, T ec hnisc he Univ ersit¨ at Darmstadt, 08.08.2008 Optimally Efficien t Prefix Searc h and Multicast in Structured P2P Net w orks T ec hnical Rep ort No. TUD-CS-2008-103 T eleco op eration Rep ort No. TK-01/08, The T ec hnical Rep orts Series of the TK Researc h Division, T ec hnisc he Univ ersit¨ at Darmstadt D. Bradler a ∗ , J. Kangasharju b , M. M ¨ uhlh¨ auser a a T eleco op eration Group, Univ ersit y of T ec hnology Darmstadt, Darmstadt, German y b Departmen t of Computer Science, Univ ersit y of Helsinki, Finnland Abstract Searc hing in P2P netw orks is fundamen tal to all o v erla y net w orks. P2P net w orks based on Distributed Hash T ables (DHT) are optimized for single k ey lo okups, whereas unstruc- tured net w orks offer more complex queries at the cost of increased traffic and uncertain success rates. Our Distributed T ree Construction (DTC) approach enables structured P2P net w orks to p erform prefix searc h, range queries, and multicast in an optimal w a y . It achiev es this b y creating a spanning tree ov er the p eers in the searc h area, using only information av ailable lo cally on each p eer. Because DTC creates a spanning tree, it can query all the p eers in the search area with a minimal n um b er of messages. F urthermore, w e sho w that the tree depth has the same upp er b ound as a regular DHT lo okup which in turn guaran tees fast and responsive run time b ehavior. By placing ob jects with a region quadtree, w e can p erform a prefix searc h or a range query in a freely selectable area of the DHT. Our DTC algorithm is DHT-agnostic and w orks with most existing DHTs. W e ev aluate the performance of DTC o v er sev eral DHTs b y comparing the p erformance to existing application-lev el m ulticast solutions, w e sho w that DTC sends 30–250% few er messages than common solutions. ∗ this researc h is funded by DFG 1 2 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser 1. In tro duction P eer-to-p eer netw orks are typically divided in to unstructur e d and structur e d net works, dep ending on how the ov erlay is constructed and how con ten t is placed in the netw ork. In an unstructured netw ork, no des are free to c ho ose their o v erla y neighbors and are free to offer any conten t they w an t. In con trast, in a structured net w ork, the structuring mec hanism (t ypically one or more hash functions) uniquely determine the lo cation of a p eer in the o v erla y and its neigh b ors, as w ell as the placemen t of con ten t on p eers. Both these kinds of netw orks ha v e their strengths and weaknesses. Unstructured net- w orks are based on se ar ching for conten t, whic h allo ws us to use complex queries for determining whic h ob jects match whic h requests. Ho w ev er, this comes at the exp ense of ha ving to flo o d the search query through the net w ork which causes significan t amount of net w ork traffic. The original Gnutella net w ork w as a completely flat o v erla y whic h relied on lo cating ob jects b y flo o ding the net work up to a given time-to-liv e (TTL). Newer un- structured netw orks, like Kazaa and eDonk ey , build a t wo-tier hierarc h y with ordinary p eers and sup erp eers. Ordinary p eers connect to one sup erp eer and the sup erp eers build a Gn utella-lik e ov erlay b etw een them. Although such a hierarchical net w ork scales b etter than the plain Gn utella net w ork, we still cannot guaran tee that an ob ject can be found and the amount of netw ork traffic caused b y flo o ding b et w een the sup erp eers can still b e high. A structured netw ork, on the other hand, clearly defines the ov erlay structure and ob ject placement through hash functions. The main adv antage is that it allows for v ery efficien t key-v alue lo okups, similar to traditional hash tables. Ho w ev er, b ecause the con ten t is placed with hash functions, real searc h queries are not feasible in DHTs. F or example, it is not feasible to query a DHT for all ob jects whose name b egins with “F o o”. This w ould usually require asking every p e er whether it has any matching ob jects. W e tac kle the problem of implementing a prefix searc h on a structured P2P netw ork. Our algorithm selects a subset of the ov erlay netw ork, and creates a spanning tree for that subset ro oted at an y of the p eers in that set. T ypical applications whic h b enefit from prefix search are all systems whic h need to deal with structured data. Structured data is v ery commonly used in man y applications. F or example storing data based on geographical lo cation (e.g., tourist information) or any kind of classification systems lend themselv es readily to our algorithm. The more attributes are a v ailable in the user pro vided searc h term (i.e., the more precise the user’s query), the less no des will b e queried. In the case of a user performing a very precise search, the query will reduce itself to a standard DHT key lo okup. Because w e create a spanning tree, our solution is not limited to implemen ting prefix searc h; other applications, suc h as m ulticast and broadcast, can also b e implemented with our algorithm. Multicast (and also broadcast) is extremely useful in cases where the DHT is built according to some sp ecific criteria (as opp osed to a standard hash function; see Section 3), since it gives us the possibility of reaching a giv en set of no des with minimal o v erhead. The k ey feature of our algorithm is that it creates a spanning tree without any c ommu- nic ation b etwe en the p e ers , using only lo cal information av ailable to every participating p eer. Because it is a tree, w e are guaran teed that only the minim um num b er of messages needs to b e sen t. These k ey features set our algorithm apart from previous w ork, such as Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 3 the application-lev el m ulticast prop osed by Ratnasamy et al. [1] or SplitStream [2]. Pre- vious work t ypically either has a high num b er of duplicate messages, requires additional co ordination traffic or needs a second ov erlay netw ork. Additionally range and origin of the spanning tree can freely b e set to an y v alue and an y p eer. Our distributed tree construction algorithm (DTC) presented in Section 2 has none of the shortcomings of the previous algorithms. It can b e used on top of most existing DHTs and it w orks using information a v ailable at each p eer ab out the p eer’s neigh b ors. As we show in Section 2, this standard information maintained b y the DHT is sufficient for spanning a tree. As our ev aluation shows, our DTC algorithm yields optimal p erformance in terms of messages sen t. W e also sho w that the o v erhead of duplicate messages in existing solutions is at least 30% but can in sev eral cases b e up to 250%. This pap er is organized as follo ws. In Section 2, w e presen t how w e construct the dis- tributed spanning tree and pro v e its prop erties. Section 3 sho ws an example of ho w to build a prefix searc h using hash functions and our tree construction algorithm. Section 4 ev aluates our algorithm on differen t DHTs and compares its p erformance against existing algorithms. In Section 5, w e discuss the robustness of our algorithm and presen t mec h- anisms for impro ving its resilience against malicious peers. Section 6 discusses related w ork. Finally , Section 7 concludes the pap er. 2. Distributed T ree Construction In unstructured net works, more than 70% of the messages are redundan t, even with a mo derate TTL [3]. This high o v erhead is one of the main reasons for the p o or scalabilit y of unstructured net works. With our distribute d tr e e c onstruction algorithm (DTC), w e are able to eliminate this o v erhead. Note that DTC builds on top of a standard, structured o v erla y net w ork. W e first discuss the requirements on the structured o v erla y , and then presen t the DTC algorithm with optimality pro ofs. Finally , in Section 2.4 w e discuss differen t applications and ho w to build them using DTC. The idea b ehind DTC is to build a spanning tree to connect all the no des we w an t to searc h. When a query is sent from the ro ot, ev ery node in the tree receives it exactly once. The challenge lies in constructing the tree without an y ov erhead and using only lo cal information a v ailable in eac h no de. 2.1. Structured Overla y Our DTC algorithm works on any structured ov erlay (a.k.a. distributed hash table) whic h fulfills the prop ert y that ev ery no de kno ws al l of its immediate neighbors in the o v erla y hash space. Net works lik e Chord [4], CAN [5] and V oroNet [6] ob viously fulfill this prop ert y . In case of Chord, the critical information is kno wing the successor and for CAN, kno wing all neighbors in all co ordinate directions. In case of Pastry [7] the condition is fulfilled since the leaf sets of all no des alwa ys contain the closest neighbors in the hash space. Although T ap estry [8] is very similar to Pastry , it do es not ha v e the equiv alen t of the leaf set and th us migh t not be suitable without modifications. Kademlia [9] also fulfills the required condition, since the buck ets for the shorter distances con tain the closest no des in the hash space and they should b e complete. In the remainder of this paper, we consider only Chord and CAN as ov erlay netw orks. 4 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser Although the principle of DTC in b oth netw orks is the same, differences in the o v erla y structures lead to p erformance differences (see Section 4). 2.2. DTC Algorithm In the distributed tree construction algorithm, we start spanning the tree from a p oin t in the o v erla y and expand from that point through the o verla y according to the ov erlay routing. The area of the ov erlay whic h the spanning tree is supp osed to co v er is explicitly defined. The information about the ro ot of the tree and the area are sufficien t to construct the spanning tree in a purely distributed manner. W e will no w show ho w this can b e done in Chord and CAN. Note that even though the algorithm constructs a spanning tree, no p eer has a complete view of the tree. The tree is alwa ys constructed on-demand by having the ro ot send a message whic h constructs the tree as it gets passed through the p eers. Thus, when w e say b elo w that a p eer adds some other p eers to the tree as its c hildren, it means in practice that the p eer in question forw ards a message to the other p eers. Example: Chord In case of Chord, the area is an arc on the Chord ring and the root of the tree is the first no de on the arc. The simplest solution is simply to add no des to the tree along the c hain of successors until the end of the arc has b een reached. How ever, this is not v ery efficien t for large areas. Instead, w e should use the fingers to create shortcuts and broaden the spanning tree. The ro ot of the tree selects all of its fingers that are in the area as its c hildren. Eac h of them will recursively p erform the same op eration un til all p eers in the area (the arc) hav e b een included in the tree. An y of the p eers can easily determine whic h of its fingers it should include, since it kno ws the ro ot and the length of the arc. The successor of the last p oint of the arc 2 is part of the tree, since the last p oin ts in the arc migh t con tain ob jects whic h are stored on their successor. Example: CAN In case of CAN, the area is a conv ex area of the d -dimensional co ordinate space, with the ro ot somewhere in this area. The restriction to conv ex search areas is imp osed by our algorithm. Non-con v ex areas can b e searc hed by splitting the searc h into non-o verlapping con v ex searches whic h co v er the desired area. The ro ot first adds its immediate neighbors (2 d neigh b ors in a d -dimensional CAN), which then con tinue adding their neigh b ors, according to the rules defined b elo w. As in the Chord-case ab ov e, the information ab out the ro ot of the tree and the area it is supp osed to co ver are a v ailable to the p eers. The area can b e defined either with simply the radius of the area, or b y sp ecifying for each dimension separately ho w far the area reaches in that dimension. As mentioned, the only restriction on the searc h area is that it m ust b e con v ex. W e assume ev ery no de kno ws the follo wing: • Size of the zone of eac h neigh b or (main tained b y standard CAN routines) • Ro ot of the tree and the area it is supp osed to cov er (av ailable in the message which is used to create the tree) 2 Often the first no de after the arc. Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 5 12345678 1 2 3 4 5 6 7 8 O BC A K L M Figure 1. Example spanning tree of a CAN Figure 1 sho ws in an example a spanning tree of a t w o dimensional CAN. The tree is ro oted at the white zone mark ed O at co ordinates (5, 5). The other white zones are the c hildren of the ro ot, and the levels of the tree are sho wn in increasingly dark er shades of gra y . The result is a spanning tree consisting of all no des within the CAN ov erlay . The tree is constructed as follo ws. When a p eer X receiv es the query , it computes for ev ery one of its neighbors the vector from the cen ter of the ro ot’s zone to the cen ter of the neigh b or’s zone. If that vector intersects the common b order surface b et w een X and the neigh b or, then X should add that neighbor as its child. The vector has to intersect the common border surface betw een the t w o no des; it is not sufficient for the vector to pass through X ’s zone. Consider the third zone from the left on the b ottom ro w in Figure 1, mark ed B . This zone has one light gra y neighbor A on the top and one medium gra y C neigh b or to the righ t. The v ector from the ro ot passes through b oth of these neighbors, but the one on the righ t (mark ed C ) is the parent of no de B . It is imp ortan t to note that ev ery no de is able to compute the v ectors and determine whether it should add an y of its neigh b ors as children (and th us forward the message) b y using only information a v ailable lo cally through normal o v erla y comm unications. No co ordination b et w een no des is needed, nor is an y additional traffic generated. 2.3. Pro of of Optimalit y W e pro v e the follo wing prop erties of our DTC algorithm: 1. The DTC algorithm creates a spanning tree o v er the area 2. The depth of the tree is prop ortional to message complexity of the underlying DHT F or the simple version of a Chord-based DTC (i.e., every no de passes the query to its successor), the first prop ert y is obvious. The proof of the first prop ert y with fingers is also straigh t-forw ard and is omitted for space reasons. In the follo wing, w e will pro v e the prop erties for a CAN-based DTC. W e make the simplifying assumption that the o v erla y net w ork is able to heal itself under ch urn without loss of messages. Note that this assumption explicitly al lows c h urn as long as c hanges 6 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser to the o verla y structure are p erformed in a lo cally atomic manner and no no de departs b et w een receiving and forw arding a query . W e also assume that the area o v er which the tree is to b e spanned is con v ex. Theorem 1. Al l no des ar e adde d to the tr e e at le ast onc e. Pr o of. F or ev ery zone Z in the conv ex area, there is a single vector whic h connects the cen ter p oints of that zone with the zone of the ro ot of the tree. Starting from zone Z , the v ector determines Y , a neighbor of Z who will add Z as its c hild. Considering zone Y , w e can draw the v ector b et w een the cen ter of Y and the starting zone, whic h determines a zone X , neigh b or of Y which adds Y as a c hild. Con tinuing in a similar manner, we arriv e at the zone of the ro ot of the tree. Th us, we are able to find a chain of zones whic h leads us from the ro ot zone to zone Z . Thus, all nodes hav e at least one path from the ro ot, i.e., are part of the tree. In some cases, it is p ossible that the vector b etw een the root and a zone Z do es not pass through any direct CAN neigh b or of Z . F or example, in Figure 1, the vector b et w een the ro ot and the zone mark ed K passes directly through the corner p oin t of the t w o zones. Dep ending on ho w the ownership of edges is defined, it is p ossible that there is no neigh b or through whose zone the v ector passes on its w a y from ro ot to K . (Note that regardless of ho w the o wnership of edges is defined, it is alw a ys p ossible to construct the zones suc h that this problem p ersists.) In general, this issue arises when tw o zones share up to ( d − 2) dimensions in a d -dimensional CAN (e.g., a p oint in 2-dimensional CAN and a p oin t or a line in a 3-dimensional CAN). In this case, the forwarding algorithm do es not reac h all no des. W e hav e defined the following tie breaker for these cases. The Tie B reak er: W e use the follo wing rule for determining ho w to construct the tree in the ab o v e case. The t w o problematic zones differ in at least 2 and up to d dimensions. W e order the dimensions beforehand. The forw arding path should be suc h that the smallest dimensions with differences are used first. The length of the tie breaker path will b e the same as the num b er of dimensions in whic h the t wo problem zones differ (i.e., b et w een 2 and d ). Note that none of the no des on the path w ould normally forw ard the query , but al l of them are able to compute lo cally that they are part of the tie breaker pro cedure and are able to p erform their duties correctly . In the example of Figure 1, the tie breaker w ould mean that M is the no de resp onsible for adding K as its child, since x-co ordinate is considered b efore y-co ordinate. Theorem 2. Al l no des ar e adde d to the tr e e at most onc e. Pr o of. W e prov e this b y contradiction. If a no de A were to b e added to the tree t wice, this w ould imply that tw o of its neighbors would think that the v ector b et w een A and the ro ot passes through their zones. This is clearly imp ossible, since the resp onsibility is defined by the vector and the v ector b etw een A ’s cen ter p oin t and the cen ter of the ro ot’s zone intersects only one of the b orders betw een A and its neigh b ors. Thus, A can b e added to the tree b y at most one of its neigh b ors. Theorems 1 and 2 pro v e that ev ery no de in the area is added to the tree exactly once, th us the DTC algorithm creates a spanning tree ro oted at the ro ot zone and cov ering all the no des in the area. Note that the spanning tree is only a spanning tree; it might not Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 7 (a) Spanning tree ov erlaid on CAN (b) Spanning tree Figure 2. Example of DTC-constructed tree b e the minimal spanning tree (but this prop erty is not a requiremen t of the applications w e are considering). Figure 2 sho ws a larger example of how the DTC algorithm constructs the spanning tree. In Figure 2(a), we sho w the spanning tree o v erlaid on the underlying CAN top ology and sho w whic h p eers add whic h other p eers as their c hildren. Figure 2(b) shows only the resulting spanning tree. Theorem 3. Depth of the sp anning tr e e is pr op ortional to message c omplexity of the underlying DHT. W e pro ve this for b oth Chord- and CAN-based DTC. In case of a Chord-based DTC, w e can construct the spanning tree using fingers, as men tioned ab ov e. The links in the Chord-DTC spanning tree are determined b y the fingers and successor p ointers of the no des. The claim of the theorem refers to the depth of the spanning tree. Note that ev ery path in the spanning tree is a legal DHT-routing path b et w een the ro ot and the c hosen no de. Pr o of for a Chor d-b ase d system. In a Chord-based system with fingers, the spanning tree is simply a mapping from the fingers and successor p ointers to the no des, and every path in the tree exactly corresp onds to the routing path that the standard Chord routing would tak e to reac h that no de. Hence, the depth of the tree is O (log ( N )). Pr o of for a CAN-b ase d system. In a CAN-based system, ev ery hop is also a legal CAN routing hop, how ever, not necessarily a hop that the standard greedy CAN routing w ould tak e in a giv en situation. The DTC algorithm alw ays follows the vectors, but the greedy CAN routing migh t tak e shortcuts o v er large zones. Nevertheless, the length of the path in the spanning tree is still O ( d √ n ). 8 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser 2.4. Sample Applications The abilit y to construct a spanning tree from an y p oin t in the DHT is v ery p ow erful, and allows us to dev elop many differen t kinds of applications. W e now discuss some of the applications whic h can b e built with DTC. Prefix Search in DHTs As men tioned in the introduction, searching in DHTs is extremely inefficien t. With DTC and the hashing sc heme from Section 3, we are able to implemen t a prefix search o v er a freely selectable prefix with only the minim um n um b er of messages needed. W e ac hiev e this through a sligh t mo dification of ho w con ten t is mapped on the nodes (see b elo w) and b y spanning a tree o v er a pre-determined area. As discussed in the in tro duction, man y applications can benefit from prefix searc hes. In particular, applications whic h use any kind of structured data lend themselves readily to prefix searches. Structured data is very common and easily maps to hierarc hical concepts whic h are used in man y differen t applications. Group Communication Primitiv es If the ro ot sends a message along the tree (as is done during the tree construction), then every no de in the area will receiv e the message exactly once. By tuning the area whic h the tree spans, we can easily define differen t multicast groups and reac h them with the minim um n um b er of messages. (As the comparison in Section 4 shows, the application-lev el m ulticast on CAN [1] has significant ov erhead compared to our DTC- based approach.) It is ev en p ossible to let the area b e the complete hash space of the DHT, in whic h case w e ha v e an optimal broadcast mec hanism. An imp ortant p oin t to k eep in mind when designing applications running with DTC is whether there will b e feedback to the ro ot of the tree or not. In other w ords, a searc h requires an answ er, i.e., all the nodes in the tree with matc hing conten t should answer. In con trast, a m ulticast or a broadcast migh t not require an y ackno wledgement from the receiv ers. The presence or absence of feedback is thus application-dep enden t, and we will return to this issue in Section 5. 3. Prefix Search in a DHT W e no w present a w a y of mapping ob jects to p eers in a DHT, such that the DTC algorithm allo ws us to p erform prefix searches on the DHT. W e achiev e this b y placing all ob jects matching a prefix in a certain area of the DHT and the prefix search corresp onds simply to spanning a tree with the DTC algorithm o v er this area. As an example of ho w to implement a prefix searc h, w e consider the case of a d -dimensional CAN. (Note that other DHTs men tioned as candidates in Section 2.1 are 1-dimensional, thus they are simply sp ecial cases of the example.) 3.1. Quad T rees The standard hashing algorithm of the DHT needs to b e replaced by a function which maps equal names to equal places. W e use a region quad tree [10] to preserve the lexical order of all keys. The result is one is able to spot in adv ance the area where a set of keys with the same prefix is lo cated. One example of suc h a mapping is sho wn in Figure 3. The hash space is divided into 4 quadran ts, according to the first c haracter of the ob ject name (in the example, ob ject names can only contain capital letters A–Z and n umbers 0–9; the Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 9 12345678 1 2 3 4 5 6 7 8 [A-I]* [J-R]* [S-0]* [1-9]* [J-R][A-I]* [J-R][S-0]* [J-R][J-R]* [J-R] [J-R] [S-0]* [J-R][1-9]* Figure 3. Example quad tree co ding approac h easily generalizes to an y character set). Each of the quadran ts is further divided in to four quadrants according to the same mapping, and so on. As a result, an y ob ject whose name starts with the prefix “JOS”, w ould b e mapped to the shaded area near the top righ t corner of the area. Com bined with the DTC algorithm, this mapping enables an efficien t prefix search in a DHT. W e simply pick an y no de at random from the area of the desired prefix and span a tree o v er the area co v ered b y the prefix. W e can also adjust the gran ularit y of the mapping in order to control how many p eers are presen t in the areas for prefixes of differen t lengths. W e call this the split factor and it works as follo ws. As example, consider the mapping shown in Figure 3. The whole hash space is divided in to 4 quadrants and each quadran t is assigned a set of letters. When ob jects are stored, we lo ok at the first c haracter of the ob ject name and select the quadran t which matc hes. Then we split that quadran t again into 4 according to the same rules and map the second c haracter of the ob ject name. Because the space gets split once p er eac h c haracter, w e call this split factor 1. T able 1 sho ws how many no des need to b e searc hed for a given prefix. (Note that w e consider prefixes as characters of ob ject names, since this w ould be how users would use a prefix searc h.) The num b ers ha v e b een calculated for a CAN of 1 million no des and uniform distribution of p eers. As w e can see, already relativ ely short prefixes of 4–5 c haracters map to a relatively small num b er of p eers, on the order of a few thousands. Already with a prefix of 10 characters, we would typically end up with only 1 p eer being resp onsible. Increasing the split factor results in a more aggressive splitting. F or example, consider a mapping of letters and digits to 6 bits (similar to Base64 enco ding [11]). F or eac h split, w e pick 2 bits and map them to one quadrant of the space. One character is 3 such 2-bit groups, and we hav e to split 3 times b efore the area of a single character prefix can b e determined. W e call this split factor 3 and T able 1 shows an example. As we can see in T able 1, increasing the split factor results in a m uc h more aggressiv e 10 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser Length of No des Share of Share of Prefix in Area Keyspace Keyspace Split factor 1 Split factor 3 0 1,000,000 100% 100% 1 250,000 25% 1.6% 2 62500 6.25% 0.02% 3 15625 1.6% 0.0004% 4 3906 0.4% 0.0001% 5 976 0.1% < 6 ∗ 10 − 6 % 6 244 0.02% 7 61 0.006% 8 15 0.001% 9 4 0.0004% 10 1 0.0001% T able 1 Split factor 1 and 1 million no des splitting. F or example, a prefix of 2 characters maps to an area of 244 p eers. W e also define split factors smaller than 1, whic h are less aggressiv e than shown in T able 1. This means that eac h splitting requires a prefix of m ultiple c haracters. F or example, splitting factor 0.5 (i.e., prefix of 2 c haracters for each split) is implemen ted as follo ws. W e arrange all the c haracters on the x- and y-axes and then the p oints in the co ordinate space define 2-character prefixes. W e simply divide this space in to 4 equal areas and use those areas for splitting. This approac h easily generalizes, th us w e can freely select the gran ularit y of the splitting according to the needs of the application. W e do not change the general o v erla y structure only the hash function is replaced. Thus already kno wn metho ds for p erformance optimization [12] and congestion con trol [13] will apply as w ell. 3.2. Range Queries Note that regardless of the splitting algorithm and split factor, the DTC algorithm can span the tree in any con v ex area of the hash space. When spanning the tree, the DTC algorithm can also sp ecify individually ho w far along eac h co ordinate axis the tree should span. By selecting the appropriate co ordinates, it is p ossible to span the tree in suc h a manner that it corresp onds to a range query ov er the ob jects in the DHT. F urther ev aluation of the practical p erformance of range queries is part of our future researc h. 3.3. Discussion A correct split factor ma y be critical for application performance. F or the case of a prefix searc h of ob jects in a DHT, w e obviously wan t to limit the search areas to as small as possible without unduly stressing peers. F or a high split factor, ev en short prefixes map to a v ery small n umber of no des and may thus result in severe load imbalances b ecause some prefixes are more p opular than others. On the other hand, a high split factor means that only a v ery small num b er of no des need to b e included in the DTC-constructed spanning tree. Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 11 Usually the requiremen ts for distributed hash tables demand a more or less equal dis- tribution of all k eys in the hashtable. On the one hand, the equal distribution balances the a verage load p er peer whic h reduces local hotsp ots; on the other, the equalit y destroys the order of all k eys. One wa y to reduce hotsp ots is to choose a smaller split factor for m ultidimensional quad trees at the cost of larger search areas. In addition, there are sev eral approaches for cac hing or load balancing of queries within the CAN o verla y [14, 15]. T echniques such as virtual no des [4] can also help alleviate the load imbalance. Ev aluating the effectiveness of differen t load balancing tec hniques is part of our future w ork. 4. Ev aluation W e now ev aluate the p erformance of our DTC algorithm and compare it against tw o other solutions. W e tested DTC o ver t w o DHTs, Chord and CAN, to show ho w its p erformance in some cases dep ends on the underlying DHT. As comparison, w e ha v e selected the application-lev el m ulticast on CAN b y Ratnasamy et al. [1] and a simple flo o ding scheme, where each no de just forw ards a message to all of its neigh b ors, except the one where it got the message from. The P2P o verla y netw orks are simulated with the PlanetSim P2P sim ulator [16] and our co de will so on b e merged in to the main PlanetSim trunk to make it a v ailable for others to test. As already rep orted by Jones et al. [17], implemen ting CAN is not alwa ys a straigh t forw ard pro cess, but we did not encounter any further problems than men tioned in [17]. W e v erified our implemen tation of the CAN netw ork b y comparing it to the results in [17] and thorough testing on our o wn, and the results were as exp ected. Our Chord implemen tation is directly based on the original w ork in [4]. Our implemen tation of the ALM algorithm of [1] did not face the race-condition mentioned in [17, 18], since w e used the cycle based approac h of PlanetSim, whic h prev en ts the men tioned race-condition. W e simulated the 4 selected systems in PlanetSim. The t wo DTC-based approaches built the spanning tree, as describ ed in Section 2. ALM from [1] and the simple flo o ding w ere let run as sp ecified. W e v aried the size of the net work (v alues selected according to Section 3 for reasonable searc h areas) and, for CAN-based systems, also the n um b er of dimensions in the CAN. All sim ulations w ere rep eated 30 times and the reported num b ers are a v erages o v er the 30 sim ulation runs. W e measured the n umber of messages received b y eac h node and the depth of the spanning tree. The first metric determines ho w (in)efficien t the mec hanism is and the second determines ho w quic kly searc h results are a v ailable. T able 2 sho ws the distribution of ho w many messages a giv en no de receiv ed in a 2000 no de CAN with 10 dimensions. (DTC-Chord w as run on a standard Chord of 2000 no des.) The table shows for each of the systems how man y no des on a v erage received the message from the ro ot. W e cut the table at 14 messages per no de and summed up all the no des that received the message 14 times or more (applies only to simple flo o ding in this case). The last ro w shows the total num b er of messages sen t in the system. Since there are 2000 no des, 2000 messages are sufficien t in the optimal case. As exp ected, the tw o DTC-based solutions do not generate an y duplicate messages. ALM p erforms relatively w ell, generating ab out 50% to o man y messages. W e return to 12 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser Messages DTC Simple ALM DTC Receiv ed CAN Floo ding Chord 0-1 2000 1 1143 2000 2-3 0 2 684 0 4-5 0 6 78 0 6-7 0 22 12 0 8-9 0 67 2 0 10-11 0 247 0 0 12-13 0 925 0 0 ≥ 14 0 7851 0 0 Sum 2000 26106 3087 2000 T able 2 Av erage n um b er of messages receiv ed p er no de the ev aluation of the o v erhead of ALM b elo w. As T able 2 sho ws, simple flo o ding has an extremely high ov erhead in terms of messages sen t. The factor-13 ov erhead sho wn is t ypical of the p erformance of simple flo o ding. The performance of the DTC-based algorithms w as as exp ected in all inv estigated parameter com binations (e.g., netw ork size, dimensions). Both of them w ere able to p erform their task alw a ys with the minimum n um b er of messages, i.e., as many messages as no des. W e also ev aluated the depth of the spanning tree, since this directly affects the time it takes to complete the op eration (e.g., searc h or broadcast). W e compared different net w ork sizes from 200 to 20000 no des and differen t dimensions in CAN (ranging from 2 to 20). Figure 4 sho ws 5- and 10-dimensional CANs with 20000 no des. In Figure 4(b) we also plot DTC-Chord. The x-axis sho ws the n umber of hops and the y-axis sho ws how man y no des were reac hed with that many hops (i.e., how man y no des are at that depth in the spanning tree). As w e can see, simple flo o ding has the smallest depth. It alw ays tak es the shortest path to each no de, since it forwards a message to all neigh b ors but to the sender. Therefore all no des are reac hed with the minim um n um b er of hops. DTC-CAN and ALM ha ve p erformance whic h is sligh tly lo wer than simple floo ding and are v ery close to eac h other. In the case of the 5-dimensional net w ork and 20000 nodes the optim um would be about 7 hops, while ALM needs 9 and DTC-CAN 8 hops in the a v erage. The greater the num b er of dimensions used for the CAN net w ork, the smaller is the difference b et w een the approaches. Differences in the 10-dimensional CAN net w ork (see figure 4(b)) with 20000 no des are almost non-existen t; b oth ALM and DTC-CAN need ab out 7-8 hops on an a v erage, while the optimal case w ould b e ab out 6 hops. DTC-Chord (sho wn only in Figure 4(b)) has p erformance similar to DTC-CAN and ALM in the 10-dimensional case. DTC-Chord is indep enden t of the n um b er of dimensions, so it would b e in the same place in Figure 4(a). The finger tables of Chord reduce the n um b er of needed hops effectively . While the CAN net work is able to b e further optimized b y using more dimensions, the optim um num b er of hops for Chord is proportional to the densit y of the finger tables. Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 13 0 500 1000 1500 2000 2500 3000 3500 4000 0 2 4 6 8 10 12 14 16 18 20 Nodes reached Hops Simple Flood D=5 ALM D=5 DTC CAN D=5 (a) 5 Dimensional CAN with 20000 no des 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 0 2 4 6 8 10 12 14 16 18 20 Nodes reached Hops Simple Flood D=10 ALM D=10 DTC Chord DTC CAN D=10 (b) 10 Dimensional CAN with 20000 no des Figure 4. Query depth 0 2000 4000 6000 8000 10000 12000 14000 0 5000 10000 15000 20000 Message overhead Nodes reached ALM D=13 ALM D=12 ALM D=11 ALM D=10 ALM D=5 (a) Absolute floo ding o v erhead within CAN 1 1.5 2 2.5 3 0 5000 10000 15000 20000 Relative overhead Nodes in network ALM D=13 ALM D=12 ALM D=11 ALM D=10 ALM D=5 DTC CAN D=10 (b) Relative flo o ding ov erhead within CAN W e in v estigated the p erformance with dimensions ranging up to 20, but w e did not observ e an y significan t improv ement in p erformance of DTC-CAN or ALM after 10 di- mensions. W e now turn to ev aluating the o v erhead of ALM, whic h was already sho wn in T able 2. W e compare ALM against DTC-CAN. Figures 5(a) and 5(b) show ho w the o v erhead ev olv es as function of net w ork size. The x-axis shows the num b er of sim ulated no des; w e started from 200 no des and simulated up to 20000 no des. The y-axis sho ws the a v erage n um b er of generated messages. W e show sev eral v ariants of ALM, each with differen t n um b er of dimensions. Note that DTC-CAN w as alw ays able to perform optimally , i.e., as man y messages as there w ere no des. Figure 5(a) sho ws the absolute ov erhead, i.e., how many unnecessary messages ALM sen t and Figure 5(b) sho ws the relativ e o v erhead compared to DTC-CAN. As the netw ork size gro ws, the relativ e ov erhead of ALM in Figure 5(b) tends to about 32%. Ho w ev er, Figure 5(a) sho ws the interesting b ehavior of ALM. F or every n um b er of dimensions, the curve has sev eral sharp corners where the ov erhead changes considerably . The reason for this is as follo ws. 14 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser The message o verhead is heavily influenced b y the absolute num b er of dimensions and no des. The more dimensions, the more no des are needed to p opulate the d -dimensional ID space uniformly . As so on as d √ n ≥ 2, the ov erhead starts to conv erge to the estimated 32% (see figure 5(b)). Below the critical threshold, the n um b er of duplicate messages steadily increases with more no des. This explains the high p eaks for the smaller net w orks in Figures 5(a) and as Figure 5(b) sho ws, the resulting o v erhead can b e up to 250%. In summary , the DTC-based approac hes ha v e the adv an tage of generating only the minim um amount of traffic, while keeping the depth of the spanning tree similar to ALM. ALM, on the other hand, has a message o v erhead of at least 32%, in man y cases up to 250%. The simple flo o ding approach turns out to b e unusable since the enormous message o verhead would cause unacceptable congestions within the o v erla y . Ho w ev er, simple flo o ding has the fastest resp onse time of all compared approac hes. 5. Robustness of DTC Because the DTC algorithm (and other similar approaches like [1, 2]) builds a tree, it is esp ecially vulnerable to defective peers, where a defectiv e p eer could either b e the result of a crash or it could b e a malicious p eer. If a peer do es not forward the message, then all its c hildren and their c hildren will not b e included in the spanning tree. Because all DHTs hav e mec hanisms in place to detect crashed p eers and reco v er from those crashes, w e do not consider suc h system failures to b e a problem. The only case a message could b e lost is if a p eer crashes b et w een receiving a message and forwarding it, whic h is extremely rare. In all other cases, w e assume that the standard DHT m ain tenance tak es precedence o v er the tree spanning messages and that in suc h cases, the o verla y will first b e healed and th us no message loss or duplication can o ccur. In the follo wing, we consider the case of a malicious p eer not forwarding the message correctly onw ards. The case of a malicious p eer mo difying the message can easily b e detected b y picking a random public/priv ate k eypair, signing the original message, and including the public key in the query . W e first ev aluated the sev erit y of this problem b y selecting a randomly generated 20000-p eer net w ork and spanning a tree with the DTC algorithm ov er the whole net w ork. The CAN net w orks in this test had 10 dimensions. W e v aried the fraction of malicious p eers and selected the malicious p eers uniformly at random from all the p eers. F or eac h case, w e measured the n um b er of p eers who w ere not part of the tree. Each parameter com bination w as rep eated 30 times and the results w e presen t are a v eraged o v er all the runs. Figure 5 shows the fraction of unreac hed p eers as a function of malicious peers for 3 systems from Section 4 (DTC-CAN, DTC-Chord, and ALM). Already 10% of malicious p eers are able to cut off on an a verage 20% of the p eers from the spanning tree. Because the DTC algorithm eliminates all duplicate messages, it is particularly vulnerable to malicious p eers, but the ALM do es not fare muc h b etter. Only when the fraction of malicious p eers is b etw een 10% and 50% do es ALM ha v e any mark ed improv ement o v er DTC-based systems. Although not shown on Figure 5, the simple flo o ding approac h is extremely resistan t against malicious p eers. Peers b ecome unreac hed only when the fraction of malicious p eers is v ery high (t ypically o v er 70–80%). There is an interesting p oin t to note ab out the effects of the underlying DHT on the Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 15 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fraction of unreached peers Fraction of malicious peers DTC CAN D=10 DTC Chord ALM D=10 Figure 5. Impact of malicious p eers p erformance of DTC-algorithms. F or the case of 1% malicious p eers, Chord-based DTC p erformed m uc h w orse than a CAN-based DTC. In case of DTC-Chord, 8.6% of the p eers w ere not reached, whereas in a DTC-CAN, only 4.2% of the no des w ere unreached. F or comparison, in ALM 3.7% of the no des w ere unreac hed for that fraction of malicious p eers. F or other fractions of malicious p eers, the difference b et ween Chord- and CAN- based DTCs w as in practice negligible, although usually the Chord-based system had the higher n um b er of unreac hed p eers. The explanation for this is that in a 10-dimensional CAN, ev en if a malicious node is near the ro ot in the spanning tree, it cannot do muc h damage. In con trast, if the longest finger of the ro ot in Chord is malicious, it is able to cut half of the spanning tree. Thus, a Chord-based DTC is sligh tly more vulnerable to malicious p eers than a CAN-based DTC. W e no w propose solutions for remedying the problems caused b y malicious p eers. The fundamen tal problem is that a single malicious p eer can eliminate the complete sub- tree ro oted at that p eer from the spanning tree. Our primary goal is th us dete cting the presence of malicious p eers, with as little ov erhead as p ossible. The mechanisms b elow are not alwa ys able to pinp oint the malicious p eer; they only aim at disco v ering malicious b eha vior in the tree. The effectiv eness of DTC greatly depends on the kind of application using DTC. If we are searc hing for ob jects stored in the DHT (as in Section 3), then the problem simplifies greatly . This is because all p eers should return a resp onse to the root of the tree indicating whether they hav e any ob jects which match the searc h. By requiring al l p eers to send a resp onse, the ro ot can easily c heck the resp onses to see whether the complete area has b een searc hed. Any malicious p eer would easily be found out with such responses. In the interest of scalability , the resp onses should b e propagated back along the tree. F urthermore, each p eer should sign its resp onse with a k ey that is tied to its zone of resp onsibilit y in the DHT and its IP address. These signatures increase the lik eliho o d of disco v ering malicious p eers, b ecause the malicious p eer would hav e to in ven t the zones of resp onsibility and IP addresses for all the p eers in its subtree. These migh t o verlap with legitimate zones from other parts of the tree, thus indicating the presence of a malicious p eer. F urthermore, the ro ot could also verify some of the leafs of the tree (using additional communications) that the resp onses were actually sent b y them. Although 16 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser this would reduce the effects of malicious p eers, in the absence of a centralized iden tit y managemen t sc heme, it cannot completely eliminate them. Another solution is to split the ro ot of the spanning tree and instead of a single tree, generate sev eral spanning trees, each cov ering a small area of the total tree and in such a manner that the union of the smaller trees co v ers the whole area. This is easy to p erform, since the DTC algorithm allo ws us to define the area freely . The adv an tage is that each individual searc h area b ecomes relatively small and allo ws for an easier detection of malicious p eers. As discussed in Section 3, a prefix length of 4 digits results in a search area of about 4000 peers. By splitting the query into 10 queries, eac h query needs to co ver ab out 400 p eers. Requiring a resp onse in such small trees do es not present a significan t o v erhead and thus malicious p eers would b e easier to detect. Note that the resp onse traffic is required by the application and is thus indep enden t of the w a y the spanning tree is created (DTC, ALM, flo o ding, etc.). The ab o ve solutions w ork w ell in the case where the tree is spanned for a pu rp ose where a resp onse is required, e.g., a searc h. In the case of a pure one-w a y tree, such as broad- cast, no natural feedbac k channel exists. Ac kno wledgemen ts, similar to the resp onses in the searc h application, would work as describ ed ab o v e, but in this case they would generate additional traffic in the net w ork. How ever, without suc h an ac kno wledgemen t mec hanism, it is not p ossible to detect malicious p eers. Thus, w e prop ose to use the same mechanism for an y kind of application. The second solution, the creation of sev eral, non-o v erlapping spanning trees, is likely to b e more efficient at detecting malicious p eers, but a full ev aluation of the prop osed mec hanisms is part of our future w ork. 6. Related W ork Flo o ding approaches for P2P net w orks in general ha v e b een extensively in v estigated [3, 19, 20]. While simple flo o ding approaches ma y apply to unstructured net w orks, DHT net w orks can tak e adv an tage of the structured neighbor lists and reduce the num b er of duplicate messages. In the case of CAN, w ork b y Ratnasam y et al. [1] implemen ted an application-level m ulticast on top of CAN. As our ev aluation shows, this approach can hav e a significan t o v erhead and even in the b est case, will hav e an o v erhead of ab out 32%. As we hav e sho wn, the ov erhead is a function of the netw ork size and CAN dimensions and in smaller searc h areas, the o v erhead can gro w considerably higher. An impro v ed version of ALM is describ ed in [18]. They reduce duplicate messages, but duplicates ma y still occur, esp ecially in the case of unev en zone sizes within the CAN o v erla y . One approac h for Range Queries within DHTs [21] is to form a tree similar to our quad tree approac h. The difference is that they explicitly use a DHT in order to store a generated tree for accessing ob jects. Our approach, in contrast, do es not use the DHT in terface for generating a tree; it directly restructures the mapping from ob ject to no des. The adv an tage is that our solution enables prefix searc hing without an y o v erhead. Another approac h for prefix searc h is presented in [22] by Joung et al. The idea is that the prefixes build a sub-hypercub e within a h yp ercub e whic h spans a binomial tree [23]. In general a binomial tree can b e tra v ersed without generating duplicate messages. On the do wnside, the h yp ercub e needs to ha v e as many dimensions as bits in the ID space. Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 17 Giv en a common 160 bit ID space, this would lead to 160 dimensions, which in turn leads to 160 neigh b ors per no de for a binomial tree. F urther ov erhead is generated, because eac h of the 2 160 bits needs to b e addressable. Therefore a prohibitevely large n um b er of logical no des are necessary in smaller netw orks. In con trast, our DTC algorithm achiev es the same optimal n um b er of messages with no overhe ad o v er a standard DHT. A different approach for prefix searching [26] uses a com bination of different o v erla y net w orks, one for file managemen t and for site (participants) managemen t. The com bina- tion of a DHT based o verla y (e.g. CAN) with another indep endent ov erlay enables prefix searc h as w ell, but at m uch higher costs. Our solution do es not require any additional main tenance costs or additional links; all required knowledge can b e calculated locally . While our work concen trates on raw address co ding and message distribution, additional p erformance optimizations applied to routing for k eyw ord search in DHT based P2P net- w orks [24, 25] can b e used with DTC, since they are complementary to our solution. W e do not alter the underlying routing sc heme and therefore all optimizations used for the DHT apply to DTC as w ell. Applying m ulti-attribute range queriers to curren t DHTs has b een done b y c ho osing Hubs [27]. The idea of Hubs, whic h are resp onsible for one attribute each is indep enden t form the underlying DHT. A h ub can b e seen as a separate ov erlay for eac h attribute. Our prefix and m ulticast approac h can also b e applied to h ubs whic h would lead to an optimized ID space for eac h attribute. On the one hand less no des would need to b e queried, on the other hand a separate ov erlay net w ork needs to b e main tained for eac h attribute. 7. Conclusion In this pap er w e ha ve presented a distributed tree construction algorithm, whic h is able to create a spanning tree ov er a part of a DHT, with no inter-node communication and using only information av ailable lo cally on eac h no de. W e hav e also presen ted a mapping based on region quad trees for mapping ob jects to no des in a DHT in suc h a manner that the DTC algorithm is able to p erform prefix searc hes on con ten t stored in the DHT. Our solution is not limited to searches, but can also handle broadcast and m ulticast communications. Our ev aluation shows that DTC p erforms as expected and the comparison to similar approac hes from literature shows that the message ov erhead of existing solutions is considerably higher than the optimal num b er of messages sent b y DTC. The depth of the spanning trees are similar in all studied cases. W e ha v e also discussed the robustness of DTC against malicious p eers and presen ted solutions for strengthening DTC. Our future inv estigation will concen trate on ev aluating more sc hemes for exploiting the structured geometry of DHT o verla ys for impro ving securit y asp ects, m ulticast p erformance, and expressiveness of search queries. W e will inv estigate cac hing mec hanisms to o vercome the potential hotsp ots which may o ccur using a location sensitiv e ob ject to no de mapping. REFERENCES 1. S. Ratnasamy , M. Handley , R. Karp, and S. Shenker, “Application-lev el m ulticast using con ten t-addressable net w orks,” In NGC , 2001. 18 D. Bradler, J. Kangasharju, M. M ¨ uhlh¨ auser 2. M. Castro et al., “SplitStream: High-bandwidth m ulticast in a co op erativ e en viron- men t,” in SOSP , 2003. 3. S. Jiang, L. Guo, and X. Zhang, “Ligh tFlo o d: an efficien t flo o ding sc heme for file searc h in unstructured p eer-to-p eer systems,” in International Confer enc e Par al lel Pr o c essing , pp. 627–635, 2003. 4. I. Stoica et al., “Chord: A scalable p eer-to-p eer lo okup service for In ternet applica- tions,” in ACM SIGCOMM , 2001. 5. S. Ratnasam y , P . F rancis, M. Handley , R. Karp, and S. Shenk er, “A scalable conten t- addressable net w ork,” in Pr o c e e dings of A CM SIGCOMM , 2001. 6. Beaumon t, O. and Kermarrec, A.M. and Marc hal, L. and Riviere, E., V or oNet: A sc alable obje ct network b ase d on V or onoi tessel lations , in IEEE IPDPS, 2007 7. A. Ro wstron and P . Druschel, “Pastry: Scalable, distributed ob ject lo cation and routing for large-scale p eer-to-p eer systems,” in IFIP/ACM International Confer enc e on Distribute d Systems Platforms (Midd lewar e) , 2001. 8. B. Y. Zhao et al., “T ap estry: A resilien t global-scale o v erla y for service deploymen t,” IEEE Journal on Sele cte d Ar e as in Communic ations , pp. 41–53, 2004. 9. P . Ma ymounk o v and D. Mazieres, “Kademlia: A p eer-to-p eer information system based on the xor metric,” in IPTPS , 2002. 10. H. Samet, “The Quadtree and Related Hierarc hical Data Structures,” ACM Com- puting Surveys (CSUR) , pp. 187–260, 1984. 11. N. F reed and N. Borenstein, RF C 2045:Multipurp ose Internet Mail Extensions (MIME) Part One: F ormat of Internet Message Bo dies , No v. 1996. 12. Dab ek, F. et al., Designing a DHT for low latency and high thr oughput , in Pro ceedings of NSDI, 2004. 13. Klemm, F. and Le Boudec, J.Y. and Ab erer, K., Congestion Contr ol for Distribute d Hash T ables , in IEEE NCA, 2006. 14. T. Stading, P . Maniatis, and M. Baker, “P eer-to-p eer cac hing sc hemes to address flash cro wds,” in IPTPS , 2001. 15. A. Rao, K. Lakshminara y anan, S. Surana, R. Karp, and I. Stoica, “Load Balancing in Structured P2P Systems,” IPTPS , 2003. 16. P . Garc ´ ıa et al., “PlanetSim: A New Overla y Netw ork Sim ulation F ramew ork,” L e ctur e Notes in Computer Scienc e (LNCS), Softwar e Engine ering and Midd lewar e (SEM) , pp. 123–137, 2005. 17. M.B. Jones, M. Theimer, H. W ang, and A. W olman, “Unexp ected complexit y: Exp e- riences tuning and extending CAN,” Micr osoft R ese ar ch, T e ch. R ep. MSR-TR-2002- 118 , 2002. 18. M. Castro et al., “An ev aluation of scalable application-lev el m ulticast built using p eer-to-p eer o v erla ys,” in INFOCOM , pp. 1510–1520, 2003. 19. Y. Cha w athe, S. Ratnasam y , L. Breslau, N. Lanham, and S. Shenk er, “Making gn utella-lik e P2P systems scalable,” in ACM SIGCOMM , pp. 407–418, 2003. 20. W. W. T erpstra, J. Kangasharju, C. Leng, and A. P . Buchmann, “BubbleStorm: Resilien t, probabilistic, and exhaustiv e p eer-to-p eer search,” in A CM SIGCOMM , 2007. 21. Sriram Ramabhadran, Joseph Hellerstein, Sylvia Ratnasamy , and Scott Shenker, “Brief announcemen t: Prefix hash tree - an indexing data structure ov er distributed Optimally Efficien t Prefix Searc h and Multicast in Structured P eer-to-P eer Net w orks 19 hash tables,” in ACM PODC , 2004. 22. Y uh-Jzer Joung and Li-W ei Y ang, “Multi-dimensional prefix searc h in p2p net works,” in Confer enc e on Pe er-to-Pe er Computing , 2006. 23. Y uh-Jzer Joung, Chien-Tse F ang, and Li-W ei Y ang, “Keyword search in dht-based p eer-to-p eer net w orks,” in ICDCS , 2005. 24. P . Reynolds and A. V ahdat, “Efficient p eer-to-p eer keyw ord searching,” Pr o c e e dings of International Midd lewar e Confer enc e , pp. 21–40, 2003. 25. O.D. Gna wali, A Keywor d-Set Se ar ch System for Pe er-to-Pe er Networks , Ph.D. thesis, Massac h ussets Institute of T ec hnology , 2002. 26. Baruc h Awerbuc h and Christian Scheideler, Pe er-to-p e er systems for pr efix se ar ch , in PODC, pp. 123–132, 2003. 27. Bharam b e, A . R. and Agra w al, M. and Seshan, S, Mer cury: supp orting sc alable multi-attribute r ange queries. , in SIGCOMM, pp. 353–366, 2004.