Design of Transport Layer Based Hybrid Covert Channel Detection Engine

International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 DOI : 10 .5121/ijasuc.20 10.1409 93 Design of T ransport Laye r Based Hybrid Covert Channel De tection Engine Anjan K #1 , Jibi Abraha m *2 , Mamatha J adhav V #3 # Departmen t Of Compute r Science a nd Engineering , M.S.Ra maiah Institute of Technolog y, Bang alore,India 1 annjankk_msrit@in.com 3 mamsdalvi@gmail.com * College o f Enginee ring Pune, India 2 jibia.comp@coep.ac.in Abstract : Computer ne twork is unpredicta ble due to informatio n warfare and is prone to variou s attacks. Such a ttacks on network c ompromise th e most important attribute, the privacy. Most o f such attacks are devised using spe cial communicatio n channel called ``Cov ert C hann el''. T he word ``Co vert'' stands fo r hidden or n on-transparent. Network Cove rt Channel is a concealed communication path within legitimate network co mmunicatio n that clearly viola tes security p olicies laid down. The non-tra nsparency in covert channel is also referred to as trapdoor. A trapdoor is unintended design within l egitimate commun ication whose motto is to leak informa tion. Sublim inal cha nnel, a variant of covert c hannel works similarly ex cept th at the trapdoo r is set in a crypto graphic algorithm. A com position o f covert c hanne l with sublimina l chann el is the ``Hybrid Covert Chan nel''. H ybrid cov ert chann el is h omogenou s or heteroge neous mixture of tw o or more variants of covert cha nnels eithe r active at same instan ce or at different insta nces of time. De tecting su ch maliciou s c hannel activity pla ys a vital role in removing th reat to the legitimate network. In this pa per, we present a study of multi-tra pdoor cov ert channels an d introduc e design of a new de tection engine for h ybrid covert chann el in tra nsport layer visualized in TCP and SS L. Keywords : Covert Chan nel, Sublimin al Chann el, Hybrid Covert Ch annel, Netwo rk Security, Trapdoo rs I. I NTRODUCTI ON Covert channel [1] [2] [3] is a malicious conversation within a legitimate network communication. Covert channels clearly violate the security policies laid do wn by the network environment allowing t he i nformation l eak to the unauthorized or u nknown r eceiver. Covert channels do not have c oncrete definition and are scenario oriented. Covertness in these channels exhibit be hav iours li ke multi-trapdoor and protocol hopped where in which channelling is not constrained to pair of communication entities. A f undamental covert ch annel can be visualised in figure 1 depicting the covert communication model employed in the covert channel with pre- shared information encoding and decoding scheme between the covert us ers. International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 94 Further a covert channel can exist between processes in opera ting system or amongst distributed objects or it can e xist wherever communication i s establi shed. Focus in this p aper is on exploiting the covertness i n rudi mentary network m odel. At this poi nt of discussion, c overt channel is associated w ith v ariety of similarly sounding terminologies like side channel or stegnographic channel or supraliminal channel . Thes e literature term s are indifferent to each other and stand on the motto of promoting covertness in different form s or scenarios in a communication model over leg itimate network. Covert channels in general exhibit some ch aracterist ics: Capacity, Noise and Transmi ssion mode [12]. Capacity of covert channel is the quantit y of covert data that has to be transmitted. Noise is the amount of disturbance th at can i nterfere with the covert data when transmitted in the network channel. Transmission mode is many times found to be synchronous but can also be asynchronous. A broad classificat ion of the covert channels is descri bed in [5]. Hybrid covert channel [5], a v ariant of covert channel is defined as hom ogeneous or heterogeneous composition of two or m ore covert channel variants existing eith er at same instance or at diffe rent instances of time. Hybrid covert channel does not ha ve concrete composition. I t is unimaginable t o completely assess number of covert channels involved in hybrid composition. Hence detection is a tedious work. Complexity adds on i f the hybrid covert channel behaves as multi-trapdoor and protocol hoppe d [9]. Hybrid covert channel here as shown in figure 2 is visu alized as a combination of simple network covert c hannel in TCP a nd subliminal channel in SSL, both being t ransport layer protocols. Figure 2 : Hybrid Co vert Channel in Tra nsport Lay er Figure 1 : Covert Cha nnel Visualizatio n International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 95 Further sections of this paper cover various detection methods and design of a system to tackle the hybrid covert channel based on t he proper detection method. Section II explore s related wor k. Section III giv es brief i nsig ht abou t various detection methodologies available in the literature. Section IV deli neates about the system design. Conclusion and future enhancements are provided in section V. II. R ELATED W ORK Extensive work has been done to devise better detection methods to detect only covert channel either on live wire or on a dataset. T he method proposed in [6] is based o n detecting covert shells by monitoring the unusual traffic in the ne twork stream. Detection in covert timing cha nnels proposed in [ 7] is based on packet inter-arrival and the whole pr ocess is modelled as Poiss on's distribution. Illegal information flows in covert channels are tracked by tracing t he Message Sequence Charts (M SC) i n [8] . T he pape r [10] emplo ys a s tatistical protocol based detection to detect hybrid covert channel based on ana lysis made on packet headers. III. D ETECTION M ETHODS Detection methods [10] for covert channels embedded i n various protocols are relatively a new area of research. Covert channel detection is to actively monitor the illegal i nform ation flow or covert channel i n the network st ream. Covert Channel Identification is to identify a couple of resources used for covert channeling, espec ially this happen in the c ase of s torag e based covert channels. Focus in the proposed work is on active monitoring the malicious activity on th e network stream and not the i dentification of resources. Various authors across the globe have categorized detection into f ollowing categories listed below: A. Signature Based Detection It involves searching specific pre-defined patterns in the network stream and w hen the pattern appears, it triggers an a larm process. Best example for kind of channel it can detect is NetCat - which is a reverse-shell com munication between the internal ne twork and a public network . B. Protocol Based Detection It involves searching the pr otocols fo r anomalies or violations while moni toring the network stream. This requires understanding the protocol specification described i n their RFC' s and detector must be knowledgeable to scan covert vulnerable fields i n the protocol he ader. The best example for channel t hat can be f ound is Covert_TCP tool which manipulates sequence number field in TCP and I P ID in IPv4 packet for the covert comm unication. C. Behavioral Based Detection It involves creation of user profiles and reference pr ofiles with respect to network st ream in a legitimate environment. These reference p rofiles are later applied to t he pr oduction environ ment for lateral comparison of real t ime user profiles with re ference profiles. Best instance is writing arbitrary data in any packet using steg nographic techniques. D. Other Approaches Other approac hes include detection based on the data mining principles like neural net work and scenario based Bayes interference. Neural network approach involves trai ning the net work for ` t ' International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 96 period until re quired accurate values t o trigger the alarm process by the detection engine. In scenario based Bayes interference, a s ystem i s setup to check whether each suspicious matched signature ( hypothetical attack) found in t he monitored data stream i s par t of a global set (symptoms). Th en use each global set to calculate, with a Bayes i nference, the probability for a known a ttack t o be on hold know ing the P (Hypothetical attack / Symptoms) probability. If the detection engine finds a suspicious sc enario whose probability value is greater than a set threshold, an alarm process is trig gered by the detection eng ine. Above categorization can also behave as either statistical or probabilistic. A statistical approach is to run t he detection engine for ` t ' hours and record an a mount of data ` d ' . T his period is called as learning period and such approach helps to increase the accuracy and also to set the threshold value for the alarm process. A probabilistic app roach is to set a probability for the specific event S that occurs after the events P , Q and R as y %. This hel ps the detection engine t o tune itself to such events in its running period. IV. D ESIGN OF COVERT C HANNEL CREATION AND DET ECTION A. Major Design Criteria Hybrid c overt channel is visualized here a s heterogeneous combination of trapdoors placed in TCP and SSL i n the transport layer. Desig n a spects with respect to TCP and SSL take different route. T CP packets can be captured from the network interface of a system physically connected to a small scale LAN. SSL payload i s pa rt of TCP packet. In order to detect the trapdoor i n each of the protocol, first let us look at the pro cess of formation of the TCP packet when an application data is sent from the application l ayer as described in figure 3. Figure 3: H ybrid Covert C hannel For mation In figure 3, word s marked i n italics refer to covert and those with normal f ont refer to legitimate process. In order to test the d etection of such trapdoors in a channe l, the channel itself needs to be constructed. Before going an y f urther with design o f the channels and the detection engine, a decision on the detection method t o be employed plays a vital role. Network prot ocol like TCP is involved and hence protocol b ased detection can be employed but security protocol like SSL demands to have m ore runs to detect the trapdoor. Hence it follows a s tatistical approach, therefore detection m ethod for hybrid covert channel is an a malg amation of protocol based and statistical based, term ed as Statistical Protocol Based Detection . International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 97 B. Designing Hybrid Covert Channel for Experimental Setup Design i ssues here t akes two different routes as discussed below. With reference to figure 3, flow of design f ollows like first subliminal channel in SSL and then the simple network covert c hannel in TCP. 1) Designing Subliminal C hannel in SSL SSL ha d wide ra nge of cipher algorithms t hat assist in secured communication. One s uc h algorithm is the DSA that provide authentication service. Subliminal channel is cr eated in DSA as per [12]. Practically this can be done in following ways • Covert user provides his random number during the signature generation pro cess. • Covert user replaces system generated public-private keys with his keys. In ei ther of both cases, t he signature component contains the subliminal activity. If private key of covert s ender is known to the cover t receiver then decoding is very simple. This can be programmed either with OpenSSL or JSSE se cure sockets. 2) Designing Simple Network Cover t Channel in TCP As the oretically explained in [5] a covert sender can place his covert data i n covert v ulnerable fields like sequence number, Flags, Ack, options, padding and reserved. Since simple network covert channel is being constructed in this wor k, f ocus is on s equence number, padding and flags fields. In order to i mplem ent t his, a covert user needs a direct a ccess to TCP packet generation pr ocess. Practically under a programm ing platform this can be im plemented in two ways: • Jpcap libraries in Java that gives direct control of the interface to the developer, here a covert user. • BSD socket in Linux where socket creation can be done i n the raw mode of operation to create custom packet and info rming t he kernel not to app end the c hecksum as this i s d one b y t he developer . C. Design of Covert Detection Engine Design flow for detecting al so takes t wo stages; one for detecting the SSL trapdoor or subliminal channel and the other for the covert channel in TCP. For TCP based covert channel, TCP packet must be available for di agnosis; this c an be done b y employing a protocol analyser or sniffer. For SSL, it assumed that covert user has replaced the original supplied keys and also the manipulation of r andom nu mber is done. In such cases, randomness test for both keys and t he random number will prove the fact that t he trapdoor is placed by the covert party. Algorithm below gives a picture of the detection process. International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 98 Algorithm for Detection Engine Step 1: Capture TCP packet s from Network Interface from user specified network device Step 2: Store the TCP packet. Step 3: Analyse the TCP header on cover t vulnerable fields Step 4: Analyse the signature in TCP payload and test the k ey against PRNG tester Step 5: Log the entries of th e covert and subliminal activ ity. Step 6: Compute the perform ance graph an d detection content computation from t he ea ch session data set. A s ingle cycle of the detection engine st arting with the packet capturing from t he i nterface, the n to detection and ba ck t o interface can be better understood w ith flow diagram depicted i n f igure 4. Figure 4: De tection Engine C ycle V. C ONCLUSION AND F UTURE E NHANCEMENTS Hard c omprom ise on confidential information i s clearly unacceptable i n presence of security measures for legitimate network. Conspiracy between communication parties is not legitimate (Covert Parties) and existence of Hybrid Covert Chan nel i s the strongest threat in communication which should be decom missioned. Conclusion is to build system to detect t he activity of hybrid covert channel in a small scale LAN. This paper has focussed on designing such a system to evaluate its performance using an experimental test bed . The future enhancements include the International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 99 performance evaluation of the system us ing t he real tim e t est bed. Also enhancem ents could be planned to cover most of the possible covert fields in TCP pack et header like acknowledgment bounce and options. VI. A CKNOWLEDGEMENT S We thank Prof. V Muralidaran, Department Head, Department of Com puter Science and Engineering, M.S.Ramaiah Institute of Technology, Bangalore for his constant support and encouragement. Anjan Koundinya thanks Lat e Dr. V.K Ananthash ayana, Erstwhile Head, D epartment of Computer Science and E ngineering, M .S.Ramaiah I nstitute of T echnology, Bangalore , for igniting the passion for rese arch. VII . REFERENCES [1] Vishal Bharti, P ractical Develo pment and Dep loyment of Covert Co mmunicatio n in IPv4, Journal on Teoretica l and Applied I nformation T echnology, Apr 2007. [2] Seb astian Za nder et.al, Covert Channels a nd Counter M easures in Co mputer Network Protocols, IEEE communication M agazine o n survey a n tutorials, Dece mber 20 07. [3] Sweety C hauhan, Anal ysis and Detectio n of Network Co vert channel, T echnical Repo rt by Depar tment of co m puter scie nce and Ele ctrical Engineeri ng,University o f Maryland B altimore Count y, Dec 2 005. [4] Enping Li , Scott Cra ver, A supralimi nal channel in a wirele ss phone applicatio n, Proceed ings o f the 11th ACM workshop on M ultimedia and security, Septe mber 07-08, 2009 , Princeton, Ne w Jersey, USA. [5] Anjan K and J ibi Abraha m, Behaviour Anal ysis of Transport Layer based H ybrid Covert Channel, CNSA 201 0, Springer-Verla g LNCS serie s. [6] Sarder Cabuk,Carla Brod ley,Clay S heilds, IP Co vert C hannel D etection, ACM T ransaction on Information a nd System Sec urity, Vol 1 2, Article 22, Apr 2009 . [7] Sarder Cabuk, Car la Br odley, Clay Sheilds, IP Covert Timing Chan nels : Des ign and Detection, CCS' 04, Oc t 2004. [8] Loïc Hélouët, Claude Jar d, Marc Zeitoun, Co vert cha nnels de tection i n pro tocols using scenarios, SPV’03, April 2003 . [9] Steffen W endzel, P rotocol Channels, HAKIN9, J un 2009. [10] Description of Dete ction Approaches at http://gra y-world.net/pro jects/pape rs/html/cctde.html [11] Description of JP cap Libraries at http://netresearch.ic s.uci.edu/ kfujii/jpca p/doc/index.html [12] Gustavus J Simmons, T he Sublimina l Channel and Digital Signat ures, Springer -Verlag, 1 998. [13] Je rry Banks et.al. Discr ete Event S ystem Simulatio n, Third edition, Pr entice Hall, Jan 2 001. [14] Description of Rando mness test su ite - JRandT ester at http://sourc eforge.net/p rojec ts/jrandtest. International J ournal of Ad ho c, Sensor & Ub iquitous Computing (IJ ASUC) Vol.1, No.4, Dece mber 2010 100 A UTHORS PROFIL E Anjan K ha s received his B.E degree fro m Visveswariah Technolog ical Univers ity, Belgaum, India in 2007 and his M.T ech degree in Dep artment of Computer Science and Engineering, M.S. Ra maiah I nstitute of T echnology, Bangalore, India. He ha s been a w arded Best P erformer P G 2010 and Rank hol der for his academic excellence. His areas of resea rch include s Networ k Secur ity and Cryptograph y, Adhoc Net works and Mo bile Co m puting . Jibi Abraha m ha s re ceived her M.S degree i n So ftware S ystems from BIT S, Rajasthan, India in 1999 and PhD degree from Visves wariah Technological University, Be lgaum, India in 2008 in the area of Net work Security. Her areas of research interests i nclude Network routing al gorithms, Cryptograp hy, Networ k Security of Wire less Se nsor Net w or ks and Algorithms D esign. Curre ntly s he is working as a P rofessor at College of En gineering, Pune, I ndia. Mamatha Jadhav V has rece ived her M.Tech degree in Co mputer Science a nd Engineering fro m Dr. Ambedkar Institute of Technolo gy, Bangalore, India. She i s currently working as a faculty in Depart ment of Computer Science a nd Engineering, M .S. Ra maiah Institute of tec hnology, Ban galore, India. Her subject interests includ e Computer Ne tworks, DB MS and Wir eless Networ ks.