Interfacing network coding with TCP: an implementation

Interf acing netw ork coding wit h TCP: an implementat ion Jay Kumar Sundararajan ∗ , Szymon Jakubcz ak ∗ , Muriel M ´ edard ∗ , Michael Mitzenmacher † , Jo ˜ ao Barros ‡ ∗ Dept. of EECS Massachusetts Institute of T echnolo gy , Cambridge, MA 0213 9, US A { jaykum ar , szym, medard } @mit.edu † School of Eng. and Appl. Sciences Harvard Uni versity , Cambridge, MA 0213 8, US A michaelm@eecs.ha rvard.edu ‡ Instituto de T elecomu nicac ¸ ˜ oes Dept. de Engen haria Electrot ´ ecnica e de Computado res Faculdade de Engenhar ia da Universidade do Po rto, Portugal jbarros@fe.u p.pt Abstract —In pre vious work (‘Network coding meets TCP’) we proposed a new protocol that interfaces network coding with TCP by means of a coding layer between TCP and IP . Unlike the usual b atch-based coding schemes, th e protocol uses a T CP- compatible sliding win dow code in combination with n ew rul es fo r acknowledging bytes to TCP that take in to account the network coding operations in the lower laye r . The protocol was presented in a theor etical framework and consider ed on ly in conjunction with TCP V eg as. In th is paper we present a real- world implementation of th is protocol that addresses sev eral important practical aspects of incorpora ting network codin g and decoding with TCP’ s window manageme nt mechanism. Further , we work with the more wid espread and p ractical TCP Reno. Our implementation significantly adv ances the goa l of designing a deployable, g eneral, TCP-compatib le pro tocol that pro vides the benefits of network codin g. I . I N T RO D U C T I O N The T ran smission Contro l Pro tocol (T CP) was originally developed for wired networks. Sin ce wired netw or ks have very little packet loss on th e links and the predo minant sour ce of loss is b uffer overflow due to congestion, TCP’ s approach of in ferring congestion fro m lo sses works well. In c ontrast, wireless ne tworks are ch aracterized b y pac ket loss on the link and intermitten t connectivity due to fading. TCP wrongly assumes the cause o f these link losses to be co ngestion, an d reduces its transmission rate u nnecessarily , lead ing to low throug hput. These pro blems of TCP in wireless networks are very we ll studied, and several so lutions ha ve been proposed (see [1] and referen ces ther ein fo r a survey). In past work we p roposed a new protoco l called TCP/NC [2] that incorpor ates network co ding inside the TCP/IP protocol stack with th e aim o f improving TCP throughp ut in wireless networks. Th e interface of TCP with n etwork cod ing ca n be viewed a s a ge neralization o f previous work co mbining TCP with Forward Era sure Correction (FEC) schemes [3]. As opposed to coding only at the source, the p rotocol of [2] also allows in termediate nodes in the ne twork to per form re-enco ding of data. It is thu s mo re gene ral than end-to- end erasure corr ection over a single path, and can therefor e, in principle, be used in m ultipath and multicast scenario s to obtain throug hput benefits. In the cu rrent work , we presen t a real-life network codin g implementatio n ba sed on the mecha nism proposed in [2]. The main contributions of this pap er ar e as follows: 1) W e explain how to add ress the practical problems that arise in making th e network c oding and decoding oper- ations comp atible with TCP’ s w indow manage ment sys- tem, such as v ariab le packet leng th, buffer mana gement, and network co ding overhead. 2) W e demo nstrate the com patibility of our protoc ol with the widely used TCP Reno; the original proposal o f [2] considered only TCP V egas. 3) W e p resent experimen tal results on th e throug hput ben - efits of the new protoco l f or a TCP conn ection over a single-hop wireless link. Although c urrently our experi- ments on ly study beh avior over a single hop, this restric- tion is no t m andator y an d the ev aluatio n of the protocol over arbitrary topolo gies will be addr essed e lse wher e. Before beginning , we explain the implications of this n ew protoco l for im proving through put in wireless networks. T here has b een a g rowing interest in ap proach es that make acti ve use of the intrin sic broad cast nature of the wireless med ium. In the technique known as oppor tunistic routing [4], a node broadcasts its packet, and if one of its neighb ors receives the packet, that node will f orward the packet downstream, thereby obtaining a diversity ben efit. If more th an one of the neighbo rs recei ve the packet, they will have to coor dinate and d ecide wh o will forward the p acket. The MORE protocol [5] proposed the u se of intra-flow network co ding in comb ination with op portun istic routing. The ra ndom line ar mix ing (c oding) of inco ming packets a t a node befo re f orwarding them downstream was sho wn to reduce the co ordin ation overhead associated with o ppor tunistic routing . Anoth er advantage is that the coding operatio n can be easily tuned to add redu ndancy to the p acket strea m to comb at erasures. Such sch emes can p otentially achiev e capacity for a multicast connection [6]. T y pical imp lementations use batches of packets instead of sliding windows, an d are genera lly therefore not compatible with TCP . ExOR uses batching to reduce the coordination overhead, b ut as mentio ned in [ 4], this in teracts b adly with TCP’ s win dow mech anism. MORE uses batchin g to perform the cod ing operation . In this c ase, the receiv er canno t ackn owl- edge the packets until an entire batch has arrived and has been successfully decoded . Since TCP perf ormanc e heavily relies o n the timely return o f ACKs , such a delay in the ACKs would affect the ro und-tr ip time calculation and thereby reduce the throug hput. Opportu nistic routing also leads to reo rdering of p ackets, which is known to interact badly with TCP , as reordering can cause duplicate A CKs, and TCP interp rets du plicate A CKs as a sign of co ngestion. The work of [2] add resses both these issues. It propo ses a TCP-co mpatible sliding window cod ing scheme in com bination with a new acknowledgment mechanism for runnin g TCP over a network coded system. Th e sender would transmit a rand om lin ear combina tion o f packets in the TCP congestion window . The new ty pe of ACK allows the receiver to acknowledge every linear combination (degree of f reedom) that is linearly in depend ent from the previously received linear combinatio ns. The rec eiv er do es no t have to wait to decod e a packet, but can send a TCP A CK fo r every degree o f freedom received, thus eliminating the problems of using batchwise A CKs. It was shown in [ 2] that if the linear combination happens over a large enou gh finite field, then every incoming ran dom linear co mbinatio n will, with h igh pro bability , gen erate a TCP A CK for the very next unacknowledged packet in ord er . This is because th e ran dom combin ations d o not have any inherent orderin g. Th e argument h olds true even when mu ltiple paths deliver th e r andom linear combin ations. Hence the use o f random linear c oding with the acknowledgmen t o f degre es of freedom can po tentially a ddr ess the TCP r eordering pr oblem for multipath opp ortunistic r ou ting schemes . By presenting an implementation of the TCP/NC p rotoco l of [2], this work provides a way of combining TCP with network -coding -based multipath opportu nistic rou ting protocols su ch as MORE. The rest of the pa per is organized as follows. Section I I summarizes the protocol propo sed in [2] and pr ovides an overview of the system mod ifications requ ired. Section s III and IV describ e the sender side and r eceiver side mod ules in detail. In section V, we discuss the parameters defined in the algorithm and h ow they affect the performan ce. Section VI presents the results obtained from the experiment. Fin ally , conclu sions and possible future work ar e presented in Section VII. I I . A N OV E RV I E W O F T H E P R OT O C O L A. The ar chitecture The TCP/NC proto col introdu ces a network co ding layer between TCP and IP in the p rotoco l stack as shown in Fig. 1, where an enco der module lies on th e sender side and a d ecoder module lies on the receiver side. Althou gh it is not shown in Sender TCP Encoder Receiver T CP Decoder IP Network Forward path ACK path Fig. 1. Ov ervie w of the ne w protocol this figur e, the system can be ge neralized to have re-e ncoding inside the network, in a manner similar to MORE [5], but above the IP layer . B. The operations The encod er buffers packets gene rated by T CP and for e very arriv al from TCP , it transmits R r andom linear co mbination s of the buffered pa ckets on av erag e, where R is the r edund ancy factor . The c ontents of a coded p acket represent a linear combinatio n of the o riginal un coded p ackets; to conve y the combinatio n req uires an add itional network coding heade r that is add ed to the coded p acket. The original uncoded packets are retained in th e enco ding buffer until an ap prop riate TCP ACK arrives f rom the receiver side. The purpose of add ing redundan cy is to separate the loss recovery asp ect f rom the cong estion contro l aspect. Losses can now be recovered with out forcin g TCP r etransmissions and the associated co ngestion window size reduction s. T he am ount of redund ancy that is added depen ds o n how lossy th e network is. For instance, a 10% loss rate means that th e rate at which equations are sent into the network should be roug hly 10% more than the rate at which packets enter the encoder from TCP . This would ensu re that the n umber of equations reaching the receiver will match the n umber of packets en tering th e en coder . On the d ecoder sid e, up on receiving a new linear comb ina- tion the d ecoder places it in a decoding b uffer , appen ds th e correspo nding coefficient vector to th e decoding matrix , a nd perfor ms Ga ussian elimination. This process help s identify the “newly seen ” pac ket (if it exists). The notion o f a node seein g a packet was defined in [2] and is repeated here for the r eader’ s conv enien ce. Packets ar e treated as vectors over a finite field F q of size q . The k th packet that the so urce generates is said to hav e an index k and is d enoted as p k . Definition 1 (Seein g a p acket) . A nod e is said to have seen a pa ck et p k if it has enough information to compute a linear combinatio n of the form ( p k + q ) , wher e q = P ℓ>k α ℓ p ℓ , with α ℓ ∈ F q for all ℓ > k . Th us, q is a linear combina tion in volving packets with indices lar ger than k . Alternately , we can say that a packet is seen if after Gaussian elimination of the co efficient matrix, the packet co rrespon ds to one of the piv ot columns (a colum n that co ntains the fir st non- zero element in som e row). The d ecoder then sends a TCP A CK to the sender requesting the first unseen packet in ord er . Thus, the A CK is a c umulative AC K like in conv entio nal TCP . The Gaussian elimination may r esult in a ne w packet being decoded . In this case, th e decoder d eliv ers this p acket to the receiver TCP . Any A CKs gen erated by the rece i ver TCP are suppressed and no t sent to the sender . These ACKs may be used for managing the decodin g buffer . C. Clean interface with TCP An im portant poin t to note is that th e introd uction of th e n ew network codin g layer does no t requ ire any chang e in the basic features of TCP . As describ ed above, the network coding layer accepts TCP packets from the sender TCP and in return deli vers regular TCP A CKs b ack to the sender TCP . On the re ceiv er side, the d ecoder d eliv ers regular TCP packets to the r eceiver TCP and ac cepts regular TCP A CKs. Ther efore, neither the TCP sender n or th e TCP receiver sees any difference lookin g downwards in the p rotoco l stack 1 . The ma in change introd uced by the pr otocol is th at the TCP p ackets from the sender are transform ed by the encoder by the network coding process. This transform ation is removed by the decoder, making it in visible to the TCP receiver . On the return path, the T CP receiv er’ s A CKs are sup pressed, and instead the dec oder gen erates regular TCP A CKs that are d eliv ered to th e sender . Again, the sender do es not n eed to be aware of the coding lay er below . Th is inter face allows the possibility that regular TCP sender and receiver end hosts can c ommun icate through a wireless network even if th ey are located beyond the wire less hosts. W e now discuss some of the p ractical issues that arise in designing an implementation of the TCP/NC protoco l co mpat- ible with re al TCP/IP stacks. These issues were previously not considered in th e idealized setting of [2]. W e show that it is possible to implemen t a T CP-aware network-co ding layer tha t has the p roperty of a clean in terface with TCP , as describ ed above. I I I . S E N D E R S I D E M O D U L E A. F orming the coding buf fer The descriptio n o f the pr otocol in [2] assumes a fixed packet length, which allows all cod ing a nd d ecoding op erations to be perfo rmed symbol-wise on the whole p acket. That is, in [2] an entire packet serves as the basic u nit of data ( i.e. , as a single unkn own), with the implicit und erstanding that the exact same op eration is bein g p erform ed on e very symbol within the packet. The main advantage of this view is that the decodin g matr ix operations ( i.e. , Gau ssian elimination) can be perfor med at the granularity of packets instead of individual symbols. Also, the ACKs are then able to be repr esented in terms of packet numbers. Finally , the coding vectors then have one co efficient for ev ery p acket, not e very symbol. No te th at the same proto col and analysis of [2] holds even if we fix the basic unit of data as a symb ol in stead of a packet. The problem is that the comp lexity will be very high as the size 1 Certai n adva nced featu res of TCP may re quire some chan ges. S ee Sectio n V -E for further discussion. of th e c oding matrix will b e related to the numbe r of sy mbols in the coding buffer , wh ich is much more th an th e nu mber of packets ( typically , a symb ol is one b yte long). In actual p ractice, TCP is a byte-stream oriented p rotocol in which ACKs are in terms o f byte seq uence nu mbers. If all packets are o f fixed length, we can still apply the packet- lev el approach , sinc e we hav e a clear and consistent map between packet sequence n umbers and by te sequence numb ers. In reality , howe ver, TCP migh t generate segmen ts of different sizes. The ch oice of how many b ytes to gro up into a segment is usually made based on the Maximum T ran smission Unit (MTU) of the network, which could vary with time. A more com mon occurre nce is that applications may use the PUSH flag option asking TCP to packetize the curr ently outstandin g bytes into a segment, even if it does not form a segment of the maximu m allowed size. In short, it is im portant to ensure that our proto col works c orrectly in spite of variable packet sizes. A closely related problem is that of rep acketization. Repack - etization, as described in Cha pter 21 of [7], re fers to the situation where a set of bytes that were assigned to tw o different segments earlier by TCP m ay later be reassigned to the same segment during retr ansmission. As a result, the g roup ing of bytes into packets under TCP m ay not be fixed over time. Both variable packet lengths and repacketization need to be addressed when imp lementing the coding protoc ol. T o solve the first pr oblem, if we have packets o f different leng ths, we cou ld elongate the shor ter packets by ap pending sufficiently many dummy z ero symbols until all packets hav e the sam e length. This will work corr ectly as long as the receiver is so mehow informe d how many zer os were appended to e ach packet. While tr ansmitting these extra d ummy sym bols will decrease the thro ughpu t, gen erally this loss will not b e significant, as packet lengths are u sually consistent. Howe ver , if we have repacketization, then we have ano ther problem , namely it is no long er possible to v iew a packet as a single un known. This is because we would not ha ve a o ne- to-one mapping be tween packets sequen ce nu mbers an d byte sequence num bers; the same by tes may n ow occur in mo re than one packet. Repacketization a ppears to destroy the convenience of perform ing cod ing an d decoding at the packet level. T o cou nter these pr oblems, we prop ose the following so- lution. The co ding operation describ ed in [2] inv olves the sender sto ring the packets g enerated b y the TCP source in a co ding buf fer . W e pre-proc ess any in coming TCP segment before adding it to the coding buf fer as follows: 1) First, any part o f the incom ing segment th at is alre ady in the buf fer is removed from the segment. 2) Next, a sep arate TCP packet is created out of each remaining contigu ous part of the segment. 3) The source and destination port informatio n is removed. It will be added later in the network coding header . 4) The packets are ap pended with suf ficiently m any du mmy zero bytes, to make them as long as the longest packet currently in the buf fer . Every resulting packet is then added to the b uffer . This pr o- TCP SubHeader Data TCP SubHeader Data TCP SubHeader TCP SubHeader Data TCP SubHeader Data p 1 p 2 p 3 p 4 p 5 Data Fig. 2. The coding buf fer cessing ensures that the pa ckets in the buffer will correspo nd to disjoint and contiguo us sets o f by tes fro m the byte stream , thereby restor ing the o ne-to-o ne correspond ence between the packet n umbers and the byte seque nce nu mbers. T he reason the p ort inf ormation is excluded from the codin g is because port inform ation is necessary for the receiver to iden tify which TCP c onnection a coded packet corre sponds to. Hence, the por t informa tion sho uld no t be in volved in the codin g. W e refer to the remainin g part of the heade r as th e TCP su bheader . Upon decod ing the packet, the receiver can find out h ow many bytes are real and how many are dummy using the S tart i and E nd i header fields in the network coding header (described below). With the se fixes in p lace, we are read y to u se the packet-level a lgorithm o f [2]. All operation s are perfo rmed on the packets in the co ding buffer . Fig ure I II-A shows a typical state of the buffer after this pr e-proce ssing. The g aps at the end of th e pa ckets corre spond to the append ed zeros. It is impo rtant to note t hat, as suggested in [2], t he TCP control packets such as SYN packet and r eset packet are allowed to bypass the coding buf fer and are directly delivered to the receiver without any coding. B. The codin g he ader A co ded p acket is c reated b y for ming a rando m linear combinatio n of a su bset o f the packets in the coding buffer . The co ding oper ations are do ne over a field of size 256 in our implementation. In this case, a field symbo l cor resdpon ds to one by te. The header of a co ded packet sh ould contain informa tion that the receiver can use to identify what is the linear combination correspon ding to the packet. W e now discuss the header structure in more detail. W e assume that the network coding header has the structure shown in Figure 3. The ty pical sizes (in bytes) of the various fields are wr itten above them. The mea ning of th e various fields are described next: • Source and destination po rt: The p ort infor mation is needed for the recei ver to identify th e coded packet’ s session. It must no t be included in the coding o peration . It is taken out of the TCP header and includ ed in the network coding header . • Base: The TCP byte sequence number of the first byte that has not been A CKed. The field is used by intermediate nodes or the decoder to decide which packets can be safely dropp ed from their buf fers without affecting reliab ility . • n: The number of packets in volved in the linear combina- tion. • S tart i : The starting by te of the i th packet inv olved in the linear combinatio n. • E nd i : The last b yte of the i th packet in volved in the linear combinatio n. • α i : The coefficient used for the i th packet involv ed in th e linear combinatio n. The S tar t i (except S tart 1 ) and E nd i are expressed r elativ e to the previous packet’ s E nd an d S tart respectively , to sa ve header space. As shown in the figure, th is header format will add 5 n + 7 bytes of overhead for the network cod ing h eader in addition to the TCP head er, where n is the numb er of packets in volved in a li nea r combinatio n. (Note that the port informatio n is not c ounted in this overhead , since it has been removed from the TCP header .) W e believe it is po ssible to reduce th is overhead by furth er op timizing the h eader structure. C. Th e coding window In th e th eoretical version of the algorithm , the sender tr ans- mits a rand om linear c ombinatio n of all p ackets in the codin g buf fer . Howe ver, as n oted above, the size of the h eader scales with the nu mber of packets in volved in the linear combinatio n. Therefo re, mixing all packets cur rently in the buffer will lead to a very large coding header . T o solve th is p roblem, w e p ropo se mixing only a constant- sized subset of the p ackets cho sen from within the cod ing buf fer . W e c all this sub set th e cod ing wind ow . The cod ing win - dow evolves as f ollows. The alg orithm uses a fixed parameter for the maximu m cod ing window size W . The cod ing window contains the packet that arrived most recently f rom TCP (which could b e a retr ansmission), and th e ( W − 1) p ackets befo re it in sequence numb er , if possible. However , if some of the ( W − 1) precedin g packets have alread y been drop ped, the n the wind ow is allowed to extend b eyond the mo st rece ntly arrived pac ket until it includes W pac kets. Note that this limit on the coding window implies that th e code is now restricted in its power to co rrect erasures and to combat reorder ing-related issues. The choice of W will thus play an imp ortant role in the per forman ce of the sche me. The correct value fo r W will depend on the length o f burst err ors that the cha nnel is expected to p rodu ce. Other factors to be considered while choosing W are discussed in Section V. In our experiment, we fixed W ba sed on trial and error . D. Buffer man agement A packet is removed fro m the coding buf fer if a TCP A CK has arriv ed requesting a byte be yo nd the last byte of that packet. If a new TCP segment arrives when th e coding buffer is full, then th e segment with th e newest set of b ytes mu st b e drop ped. This may not al ways be the ne wly arri ved se gmen t, for instance, in the case of a T CP retransmission of a previously drop ped segment. B a s e n E n d α S t a r t E n d α … 1 4 2 1 2 2 1 Source Dest 2 2 TCP SubHeader Data TCP SubHeader TCP SubHeader Data p 1 p k p n Data Base = first un-ACK ed byte Start k End k RLC = α 1 p 1 + α 2 p 2 + α 3 p 3 + α 4 p 4 + α 5 p 5 4 S t a r t Base n End 1 α 1 Start 2 End 2 α 2 … n times Source Port Dest Port Obtained fr om TCP header Coding coef ficient Start 1 Fig. 3. The network coding header I V . R E C E I V E R S I D E M O D U L E The deco der module’ s opera tions are outlined b elow . The main d ata structur e inv olved is the deco ding ma trix, which stores the coefficient vectors cor respond ing to the linear com- binations currently in the decoding buf fer . A. Acknowledgment The rece i ver side module stores the incom ing linear com- bination in th e de coding buffer . Then it unwraps the cod ing header an d ap pends the n ew co efficient vector to th e deco ding matrix. Gaussian e limination is per formed and the packet is dropped if it is n ot in novati ve (i. e. if it is not line arly indepen dent o f pr eviously received lin ear comb inations). After Gaussian elimin ation, the oldest unseen packet is iden tified. Instead o f ack nowledging the packet n umber as in [2], the decoder acknowledges the last seen packet by requesting th e byte sequen ce number of the fi rst byte of the first un seen packet , using a r egular TCP ACK. Note that this could happen b efore the packet is decod ed an d delivered to th e receiver TCP . The port and IP add ress informa tion for sen ding this ACK may be o btained fro m the SYN packet at the beginnin g of the connectio n. Any ACKs gen erated by the recei ver TCP are not sent to the sen der . T hey are instead u sed to u pdate the receive window field that is u sed in the TCP ACKs generated by th e d ecoder ( see subsection below). They are also used to keep track of which bytes have b een deliv ered , for buffer managem ent. B. Decoding and delivery The Gaussian elimination operation s ar e p erform ed not on ly on the deco ding co efficient matrix, but co rrespon dingly also on the cod ed packets th emselves. When a new p acket is decoded, any dummy zero symb ols that wer e added by the encode r are pruned using the codin g header information. A new TCP packet is created with th e newly decoded data and the appr opriate TCP header fields and this is then delivered to the receiver TCP . C. B uffer m anagement The deco ding buf fer need s to store packets that have n ot yet be en de coded an d d eliv ered to the T CP recei ver . Deliv ery can b e c onfirmed using the receiver TCP’ s A CKs. In ad dition, the buf fer also need s to stor e those p ackets that h av e been delivered but hav e not y et been dropped by the encoder from the coding buffer . Th is is b ecause, such packets may still be in volved in in coming lin ear com bination s. The B ase field in the coding header addr esses this issue. B ase is the oldest byte in the co ding buffer . Th erefor e, the decoder ca n drop a packet if its last by te is smaller than B ase , and in addition, ha s be en delivered to and A CK ed b y the receiver T CP . Wh enever a new linear combinatio n arrives, the value of B ase is updated from the header, and any pac kets that can be dro pped ar e dropp ed. The buffer m anagemen t can be u nderstoo d using Fig. 4. It shows the receiv er side windows in a typ ical situation. In this case, B ase is less th an the last delivered byte. Hence, some delivered packets h av e no t yet been drop ped. There co uld also be a c ase where B a s e is beyond the last deli vered byte, po ssibly because nothing has been decoded in a while. D. Modifying the r eceive window The T CP receive window header field is used by the receiver to inform the sender how many bytes it can accept. Since the receiver TCP’ s A CKs are supp ressed, th e deco der must copy this infor mation in the AC Ks that it sends to th e sender . Howe ver , to ensu re corr ectness, we m ay have to mo dify the value of the TCP receive window based on the decod ing buf fer size. The last accep table byte sho uld thu s be the minimu m of the receiver TCP’ s last accep table byte and th e last by te that the decodin g buffer can accommo date. No te that while calculating the space lef t in the decoding buffer , w e can inc lude the space occupied by d ata that h as already been delivered to the re ceiv er because such data will get dr opped when B ase is upd ated. If window scaling op tion is used b y TCP , this needs to be noted Base Dropped Receive wi ndow ACKed (First unseen byte) S e e n ACKed by Receiver T CP Delivered Seen Fig. 4. Recei ver side window management from the SYN packet, so th at th e mo dified value of the receive window can be correctly rep orted. Idea lly , we would like to choose a large enoug h decoding b uffer size so that the deco ding buf fer would n ot be the bottlen eck an d th is modification would never be needed. V . D I S C U S S I O N O F T H E P R A C T I C A L I T I E S A. Redun dancy factor The choice of redund ancy factor is based on the ef fective loss probab ility on the links. For a lo ss ra te of p e , with an infinite window W and using TCP V egas, the theo retically optimal value of R is 1 / (1 − p e ) , a s shown in [2]. Th e basic idea is that of the coded packets that are sent in to the network, o nly a fraction (1 − p e ) of them a re delivered on average. Hence , the value of R m ust be chosen so that in spite of these lo sses, the receiver is able to collect linear equations at the same rate as the rate at wh ich the unk nown packets a re mixed in them by the encoder . As discussed below , in practice, the value of R may depend o n the co ding window size W . As W d ecreases, the erasure correction capability o f the code goes down. Hence, we may need a larger R to compensate and en sure that the lo sses are still masked from TCP . Another factor that af fects the choice of R is th e use of TCP Ren o. The TCP Reno m echanism causes the transmission ra te to fluctu ate aro und the link capacity , and this lead s to some ad ditional losses over an d above the link losses. Th erefor e, the optima l choice of R may be higher than 1 / (1 − p e ) . B. Coding W ind ow Size There a re se veral con siderations to keep in m ind while choosing W , the coding wind ow size The m ain idea behind coding is to mask the losses on the channel fr om TCP . In other words, w e wish to cor rect losses withou t relyin g on the A CKs. Consider a case where W is just 1 . Th en, this is a simple repetition co de. Every p acket is repeated R times o n average. Now , suc h a repetition would be usefu l only for recovering on e packet, if it was lo st. Instead, if W was say 3, then every linear combinatio n w ould be usefu l to recover any of the three packets in volved. Ideally , the linear comb inations generated sho uld be able to co rrect the loss of any of the packets that have n ot yet been AC Ked. For this, we need W to be large. This may be difficult, since a large W would lead to a large coding hea der . Another penalty of choosing a large v alue of W is related to the interaction with TCP Ren o. Th is is discussed in the next subsection. The p enalty o f keeping W small on the other h and, is that it reduces the er ror correction cap ability o f the co de. For a loss probab ility of 10%, the theoretical v alue of R is aroun d 1.1. Howe ver , th is assumes tha t all linear com binations are useful to correct any pac ket’ s loss. The restriction o n W means that a coded packet can be used on ly fo r recovering tho se W packets that have been mixed to form that coded packet. In particular, if there is a contig uous burst of losses that result in a situation where the receiver h as received no linear combina tion in volving a particu lar origina l packet, then th at p acket will show up as a loss to TCP . T his co uld ha ppen even if the value of R is chosen accordin g to the theoretical value. T o compen sate, we may have to choose a larger R . The connection between W , R and the losses that are visible to TCP can be visualized as follows. Im agine a p rocess in which whenev er the receiver re ceiv es an inn ovati ve linear combinatio n, o ne imagin ary token is gen erated, and whe never the sender slides the coding window forward by one packet, one token is used u p. If th e sender slides the coding win dow fo rward when there are no tokens left, th en th is leads to a packet loss that will be visible to TCP . The reason is, when this happ ens, the deco der will not be a ble to see the very n ext unseen packet in order . I nstead, it will skip one packet in the sequence. This will make the d ecoder gener ate du plicate ACKs requesting that lost (i.e., unseen) packet, thereby causing the sender to notice the loss. In this process, W correspon ds to the i nitial num ber of tokens av ailable at the sender . Thu s, when the difference between the number of redundant packets (linear equatio ns) received and the number of orig inal packets (unknowns) inv olved in the coding up to that point is less than W , the losses will be masked from TCP . Howe ver , if this difference exceeds W , the losses will no longer be m asked. A theor etically optimal value of W is not known. Howe ver , we expec t that the value sh ould be a functio n of the loss probab ility of the link. For th e experimen t, we cho se values of W based on trial and error . C. W orking with TCP Reno By adding enough redund ancy , the coding operation essen- tially converts the lossiness of the channe l into an extension of the roun d-trip tim e (R TT). This is why [2] pro posed the use of the idea with TCP V egas, since TCP V egas contro ls the con gestion window in a smoo ther manne r using R TT , compare d to th e mor e ab rupt loss-based variations of TCP Reno. However , the coding mech anism is also compatible with TCP Reno. T he choice of W pla ys an im portan t role in ensuring this compatibility . T he choice o f W controls the p ower of th e underly ing code , and hen ce d etermines when losses are visible to TCP . A s explained above, losses will be ma sked f rom TCP as long as the nu mber of received eq uations is no more than W short of the number o f unknowns in volved in them. For compatibility w ith Reno, we need to m ake sur e that whenever the sending rate exceed s th e link capacity , th e resultin g queue drops are v isible to TCP as losses. A very large value o f W is likely to ma sk e ven these congestion losses, thereby temporarily giving TCP a false estimate of capacity . Th is will eventually lead to a timeout, and will affect thr oughp ut. The value o f W should therefo re be large enou gh to mask the link losses and sm all enough to allow TCP to see the queue drops d ue to congestion . D. Computation al overhead It is important to implement the encoding an d deco ding operation s efficiently , since any time spent in these op erations will affect the roun d-trip tim e perceived by T CP . The finite field operations over GF(25 6) have been optimized u sing the approa ch of [8], which pr oposes the u se of log arithms to multiply elements. Over GF(256), each symbol is one byte lo ng. Addition in GF(2 56) c an be imp lemented easily as a bitwise XOR of the two bytes. The main computation al overhead o n the en coder side is the formation of the ran dom linear combin ations of the buf fer ed packets. The manag ement of the buffer also requires some computatio n, but this is small co mpared to the rando m linear coding, since the coding has to be don e on every by te of the packets. T ypically , packets have a length L of ar ound 1500 bytes. For every lin ear combinatio n that is created, the coding operation inv olves LW multiplicatio ns and L ( W − 1) additio ns over GF (256) , where W is the cod ing window size. Note that this has to be do ne R times on average for every pac ket generated by TCP . Since the code d packets are newly cr eated, allocating memory for them could also take tim e. On the d ecoder side, th e main o peration is the Gaussian elimination. No te that, to identify wh ether an incoming line ar combinatio n is innovati ve o r no t, we need to perfo rm Gaussian elimination only on the deco ding matrix, and not on the cod ed packet. If it is in novati ve, then we per form the row transf or- mation ope rations of Gaussian eliminatio n on th e cod ed packet as well. This require s O ( LW ) multiplications and additions to zero out the pi vot colu mns in the ne wly added ro w . Th e complexity of the next step of zer oing ou t the newly f ormed piv ot co lumn in the existing r ows of the decod ing matrix varies depend ing on the cu rrent size and structure o f th e ma trix. Up on decodin g a new packet, it needs to be packaged as a TCP packet and delivered to the r eceiver . Since this re quires allocating space fo r a new packet, this could also be expensive in terms of time. As we will see in th e next section, th e benefits brough t by the erasure correction b egin to o utweigh the overhead o f the computatio n and coding head er for loss rates o f a bout 3 %. T his could be improved f urther by more efficient implementation of the encod ing and deco ding op erations. E. Miscellaneous considerations The TCP/NC pro tocol require s n o modificatio n in the basic features of the TCP p rotoco l o n either the sender side or the receiver side. However , other special f eatures of TCP tha t make use o f the A CKs in ways other than to report the n ext req uired byte sequence numb er , will n eed to be handled carefully . For instance, implemen ting the timestamp op tion in the presence of ne twork cod ing across packets may require some th ought. W ith TCP/NC, the receiver may send an A CK for a packet ev en b efore it is d ecoded. Thu s, the r eceiver m ay not have access to the timestamp of the packet when it sends the AC K. Similarly , the TC P checksum field has to b e dealt w ith carefully . Since a TCP packet is ACK ed e ven before it is decoded, its checksum cannot be tested befor e AC King . One solution is to implement a separate ch ecksum at the network codin g layer to detect er rors. In the same way , the v ario us other TCP optio ns that are av ailable have to be implemented with care to ensure that they are n ot affected by the prema ture A CKs. V I . R E S U LT S W e test the protoco l on a TCP flow running over a single-hop wireless link. Th e transmitter an d receiver are Linux machines equippe d with a wireless anten na. The experimen t is per formed over 802. 11a with a bit-r ate of 6 Mbps and a maximum of 5 link layer retransmission attempts. R TS-CTS is disabled. Our implementation uses the Click modular router [9]. In order to con trol the param eters of the setup, we use the predefined elements o f Click. Since the two machines are physically close to each other, there are very few losses on the wireless lin k. In stead, we ar tificially indu ce packet losses using the R a ndomS ampl e element. Note th at these packet losses are in troduce d befor e the wireless lin k. Hence, th ey will not be recovered b y the link layer retransmissions, and have to be corr ected by the layer ab ove IP . The ro und- trip delay is empirically observed to be in the range of a fe w tens of milliseconds. The encoder and decod er q ueue sizes are set to 100 packets, and the size o f the bottleneck queue just in fro nt of the wireless link is set to 5 packets. In our setup, the loss inducing element is placed befor e the bottlenec k qu eue. The q uantity measured dur ing th e expe riment is the g oodp ut over a 20 secon d long TCP session. The goo dput is measured using iper f [10]. Each point in the plots shown is averaged over 4 or more iterations of such sessions, depending on the variability . Occasionally , when the iteration d oes no t terminate and the conne ction times out, the correspo nding iteration is neglected in the average, fo r both T CP and T CP/NC. This happen s aroun d 2 % of the time, and is o bserved to be because of an un usually long burst of losses in the forward or retu rn path. In the comp arison, neither TCP nor TCP/NC uses selecti ve A CKs. TCP uses d elayed A CKs. Howe ver, we have not im plemented delayed A CKs in TCP/NC at th is poin t. Fig. 6 sho ws the variation of the g oodpu t with the redund ancy factor R f or a loss r ate of 1 0%, with a fixed coding window size of W = 3 . The theoretically op timal value of R for this loss rate is 1 .11 (=1/0.9 ) (see [2]). Howe ver, from the experiment, we find that the best goodp ut is achieved for an R o f ar ound 1.2 5. The discr epancy is po ssibly because of the type o f cod ing scheme employed . Our c oding sch eme tr ansmits a linear co mbination o f only the W most recent arri vals, in order to save packet header space. This restriction reduces the strength of the code f or the same v alue of R . In general, the value o f R and W must b e carefu lly chosen to get th e best 0 5 10 15 20 25 0 100 200 300 400 500 600 Packet Loss Rate (%) Goodput (in kilobytes/s) TCP TCP/NC Fig. 5. Goodput versus loss rate benefit of the coding operatio n. As mention ed ear lier , nother reason for the discrepancy cou ld be the use of TCP Reno. Fig. 7 plots th e v ariatio n of goo dput w ith the size of the coding win dow size W . The lo ss rate for this plot is 5%, with the redun dancy factor fixed at 1.06 . W e see th at the b est c oding window size is 2. Note that a coding window size of W = 1 correspo nds to a repetition cod e th at simply transmits ev ery packet 1.06 times on av erag e. In comparison, a simp le sliding window co de with W = 2 brings a big gain in throug hput by making the a dded redun dancy m ore u seful. Howe ver , goin g beyond 2 redu ces th e go odput because a large value of W can mislead TCP into belie ving that the cap acity is larger than it really is, which leads to timeouts. W e find that the b est value of W for our setup is usually 2 fo r a loss rate up to aro und 5 %, and is 3 fo r higher loss ra tes u p to 25%. Besides th e loss rate, the value of W could also depend on other factors such as the roun d-trip time of the path . Fig. 5 shows th e goodpu t as a function of the packet loss rate. For each loss rate, the v alue s of R and W have been ch osen by trial an d err or, to be th e one that maximizes the g oodp ut. W e see that in th e lossless case, TCP perfo rms better than TCP/NC. Th is co uld be bec ause of th e computatio nal overhead that is introduced by th e co ding and d ecoding ope rations, and also the co ding he ader overhead. However , as th e loss rate inc reases, the b enefits of coding begin to o utweigh th e overhead. The goo dput of TCP/NC is ther efore h igher than TCP . Coding allows lo sses to b e masked from TCP , and hence the fall in g oodp ut is m ore gradual with coding than withou t. The perform ance can be impr oved further by improving the efficiency of the computatio n. V I I . C O N C L U S I O N A N D F U T U R E W O R K In this paper , we ha ve demon strated experimentally , a ne w sliding-wind ow coding scheme that is compatible with TCP- Reno. The scheme allows the interfacing of TCP with network coding. This has implications for run ning TCP over wireless 1.1 1.15 1.2 1.25 1.3 1.35 1.4 1.45 200 250 300 350 400 450 Redundancy factor (R) Goodput (in kilobytes/s) Loss rate = 10%, Coding window size (W) = 3 Fig. 6. Goodput versus redundanc y factor for a 10% loss rate and W=3 1 2 3 4 5 0 50 100 150 200 250 300 350 400 450 Coding window size (W) Goodput (in kilobytes/s) Loss rate = 5%, Redundancy factor (R) = 1.06 Fig. 7. Goodput versus coding window s ize for a 5% loss rate and R=1.06 networks, in particu lar in the context of lossy multipath oppor- tunistic routing scenario s. W e believe that the proposed ideas and the imp lementation will lead to the prac tical realization of the theoretically promised benefits of network coding in such scenarios. This end eav or would requ ire more work in the futu re, in terms of under standing the r ole p layed by the various param e- ters of the ne w pr otocol, for instance, the red undan cy factor R and the coding window size W . T o achieve high thro ughpu ts in a fair manner, the v alue s of R and W have to be caref ully adapted based on the characteristics of the underly ing chan nel. In th e futu re, it would b e usefu l to extend th is imp lementation to a mu lti-hop network with multiple paths from the sender to the receiv er and re-encod ing of packets inside the network. A C K N OW L E D G M E N T S The authors would like to thank Pro f. De va vrat Sh ah and Prof. Dina Katabi f or several useful discussions. W e would a lso like to thank Mythili V utukur u and Rahu l Ha riharan for their help and advice regarding the imp lementation . R E F E R E N C E S [1] S. Rangwal a, A. Jindal, K.-Y . Jang, K. Psounis, and R. Govindan, “Understa nding congesti on control in multi-hop w ireless mesh networks, ” in Pr oc. of ACM/ IEEE Internati onal Conf erenc e on Mobile Computin g and Networki ng (MobiCom) , 2008. [2] J. K. Sundarara jan, D. Shah, M. M ´ edard, M. Mitz enmacher , and J. Barros, “Netw ork coding meets TCP, ” in Proc eedings of IEEE INFOCOM , April 2009, pp. 280–288. [3] F . Brockners, “The case for FEC-fueled TCP-like congesti on control, ” in K ommunikati on in V erteilten Systemen , ser . Informatik Aktuell, R. Stein- metz, Ed. Springer , 1999, pp. 250–263. [4] S. Biswa s and R. Morris, “ExOR: opport unistic multi-hop routi ng for wireless netw orks, ” in Proce edings of ACM SIGCOMM 2005 . A CM, 2005, pp. 133–144. [5] S. Chachu lski, M. Jennings, S. Katti, and D. Katabi , “T rading structure for randomness in wireless opportu nistic routing, ” in Pro c. of ACM SIGCOMM 2007 , August 2007. [6] T . Ho, M. M ´ eda rd, R. Koet ter , D. Karger , M. Effros, J. Shi, and B. Leong, “ A random linea r network coding approach to multicast , ” IEE E T rans. on Informatio n Theory , vol. 52, no. 10, pp. 4413–4430, October 2006. [7] W . R. Ste vens, TCP/IP Illustrate d, V olume 1: The Protoc ols . Addison- W esley , 1994. [8] N. R. W agner , The Laws of Cryptograp hy with J ava Code . [Online]. A vai lable: http:// www .cs.utsa.edu/ ∼ wagne r/lawsbookc olor/laws.pdf [9] E. K ohler , R. Morri s, B. Chen, J. Jannotti, and M. F . Kaashoek, “The Cli ck modular route r, ” in ACM T ransactions on Computer Systems , vol . 18, no. 3, August 2000, pp. 263–297 . [10] NLANR Distrib uted Applica tions Support T eam, “Iperf - the tcp/udp bandwi dth measurement too l. ” [Online]. A vailab le: http:/ /dast.nlanr .net/Projects/Iperf/