Security Analysis of Online Centroid Anomaly Detection

Securit y Analysis of Online Cen troid Anomaly Detection Marius Kloft ∗ mkloft@cs.berkeley.edu Computer Scienc e Division University of California Berkeley, CA 94720-1758, USA P a vel Lask o v p a vel.lasko v@uni-tuebingen.de Wilhelm-Schickar d Institute for Computer Scienc e University of T¨ ubingen Sand 1, 72076 T¨ ubingen, Germany Abstract Securit y issues are crucial in a num b er of mac hine learning applications, esp ecially in scenarios dealing with human activit y rather than natural phenomena (e.g., information ranking, spam detection, malw are detection, etc.). It is to b e expected in such cases that learning algorithms will hav e to deal with manipulated data aimed at hamp ering decision making. Although some previous w ork addressed the handling of malicious data in the con text of supervised learning, v ery little is kno wn about the b eha vior of anomaly detection metho ds in suc h scenarios. In this con tribution, 1 w e analyze the performance of a particular metho d – online centroid anomaly detection – in the presence of adv ersarial noise. Our analysis addresses the following securit y-related issues: formalization of learning and attac k pro cesses, deriv ation of an optimal attack, analysis of its eﬃciency and constraints. W e deriv e b ounds on the eﬀectiveness of a p oisoning attack against centroid anomaly under diﬀeren t conditions: b ounded and unbounded percentage of traﬃc, and b ounded false p ositiv e rate. Our b ounds show that whereas a p oisoning attac k can b e eﬀectively staged in the unconstrained case, it can b e made arbitrarily diﬃcult (a strict upp er b ound on the attac ker’s gain) if external constraints are prop erly used. Our exp erimental ev aluation carried out on real HTTP and exploit traces conﬁrms the tightness of our theoretical bounds and practicality of our protection mec hanisms. 1. In tro duction Mac hine learning methods ha v e b een instrumen tal in enabling numerous no vel data analysis applications. Currently indisp ensable technologies such as ob ject recognition, user prefer- ence analysis, spam ﬁltering – to name only a few – all rely on accurate analysis of massive amoun ts of data. Unfortunately , the increasing use of machine learning metho ds brings ab out a threat of their abuse . A convincing example of this phenomenon are emails that b ypass spam protection tools. Abuse of mac hine learning can tak e on v arious forms. A ∗ . Also at Machine Learning Group, T echnisc he Univ ersitt Berlin, F ranklinstr. 28/29, FR 6-9, 10587 Berlin, German y . 1. A preliminary v ersion of this pap er app ears in AIST A TS 2010, JMLR W orkshop and Conference Pro- ceedings, 2010. 1 malicious part y ma y aﬀect the training data, for example, when it is gathered from a real op eration of a system and cannot b e manually veriﬁed. Another p ossibilit y is to manip- ulate ob jects observ ed b y a deploy ed learning system so as to bias its decisions in fav or of an attack er. Y et another wa y to defeat a learning system is to send a large amoun t of nonsense data in order to pro duce an unacceptable n umber of false alarms and hence force a system’s op erator to turn it oﬀ. Manipulation of a learning system may th us range from simple cheating to complete disruption of its op erations. A p oten tial insecurity of machine learning metho ds stems from the fact that they are usually not designed with adversarial input in mind. Starting from the mainstream compu- tational learning theory (V apnik, 1998; Sc h¨ olk opf and Smola, 2002), a prev alen t assumption is that training and test data are generated from the same, ﬁxed but unknown, probabil- it y distribution. This assumption ob viously do es not hold for adv ersarial scenarios. F ur- thermore, ev en the recent w ork on learning with diﬀering training and test distributions (Sugiy ama et al., 2007) is not necessarily appropriate for adversarial input, as in the latter case one must account for a sp eciﬁc worst-case diﬀerence. The most imp ortant application ﬁeld in which robustness of learning algorithms against adv ersarial input is crucial is computer securit y . Modern securit y infrastructures are facing an increasing professionalization of attacks motiv ated b y monetary proﬁt. A wide-scale deplo yment of insidious ev asion techniques, suc h as encryption, obfuscation and p olymor- phism, is manifested in an explo ding div ersity of malicious softw are observed b y securit y exp erts. Mac hine learning metho ds oﬀer a p o werful to ol to coun ter a rapid evolution of securit y threats. F or example, anomaly detection can identify un usual even ts that p oten- tially con tain nov el, previously unseen exploits (W ang and Stolfo, 2004; Rieck and Lask o v, 2006; W ang et al., 2006; Rieck and Lask ov, 2007). Another typical application of learning metho ds is automatic signature generation whic h drastically reduces the time needed for a pro duction and deploymen t of attack signatures (Newsome et al., 2006; Li et al., 2006). Mac hine learning methods can also help researchers to better understand the design of mali- cious softw are by using classiﬁcation or clustering techniques together with sp ecial malware acquisition and monitoring to ols (Bailey et al., 2007; Rieck et al., 2008). In order for machine learning metho ds to b e successful in security applications – and in general in an y application where adversarial input ma y b e e ncoun tered – they should b e equipp ed with coun termeasures against potential attacks. The current understanding of securit y prop erties of learning algorithms is rather patc hy . Earlier w ork in the P AC- framew ork has addressed some scenarios in whic h training data is delib erately corrupt (Angluin and Laird, 1988; Littlestone, 1988; Kearns and Li, 1993; Auer, 1997; Bschout y et al., 1999). These results, how ever, are not connected to mo dern learning algorithms used in classiﬁcation, regression and anomaly detection problems. On the other hand, several examples of eﬀective attac ks ha ve b een demonstrated in the context of sp eciﬁc security and spam detection applications (Lowd and Meek, 2005a; F ogla et al., 2006; F ogla and Lee, 2006; P erdisci et al., 2006; Newsome et al., 2006; Nelson et al., 2008), whic h has motiv ated a recent w ork on taxonomization of suc h attacks (Barreno et al., 2006, 2008). Ho w ever, it remains largely unclear whether machine learning metho ds can b e protected against adversarial impact. W e b eliev e that an unequivocal answ er to the problem of “security of mac hine learning” do es not exist. The securit y prop erties cannot b e established exp erimen tally , as the notion 2 of security deals with ev ents that do not just happ en on av erage but rather only p otentially ma y happ en. Hence, a theoretical analysis of mac hine learning algorithms for adv ersarial scenarios is indisp ensable. It is hard to imagine, how ev er, that such analysis can oﬀer meaningful results for any attack and any circumstances. Hence, to b e a useful guide for practical applications of machine learning in adv ersarial environmen ts, suc h analysis m ust address sp e ciﬁc attacks against sp e ciﬁc le arning algorithms . This is precisely the approach follo wed in this con tribution. The main fo cus of our w ork is a security analysis of online centroid anomaly detection against the so-called “p oisoning” attacks. The centroid anomaly detection is a very simple metho d which has b een widely used in computer security applications (e.g., F orrest et al., 1996; W arrender et al., 1999; W ang and Stolfo, 2004; Riec k and Lask ov, 2006; W ang et al., 2006; Riec k and Lasko v, 2007). In the learning phase, centroid anomaly detection computes the mean of all training data p oin ts: c = 1 n n X i =1 x i . Detection is carried out by computing the distance of a new example x from the cen troid c and comparing it with an appropriate threshold: f ( x ) = ( 1 , if || x − c || > θ 0 , otherwise. Notice that all operations can b e carried out using kernel functions – a standard tric k kno wn since the kernel PCA (Sc h¨ olk opf et al., 1998; Sha we-T aylor and Cristianini, 2004) – whic h substan tially increases the discriminativ e p ow er of this metho d. More often than not, anomaly detection algorithms are deploy ed in non-stationary en- vironmen ts, hence need to b e regularly re-trained. In the extreme case, an algorithm learns online b y up dating its hypothesis after ev ery data point it has receiv ed. Since the data is fed in to the learning phase without any v eriﬁcation, this op ens a p ossibility for an adversary to force a learning algorithm to learn a represen tation suitable for an attack er. One particular kind of attac k is the so-called “p oisoning” in whic h sp ecially crafted data p oints are injected so as to cause a h yp othesis function to misclassify a given malicious p oint as b enign. This attac k makes sense when an attack er do es not hav e “write” p ermission to the training data, hence cannot manipulate it directly . Therefore, his goal is to tric k an algorithm by merely using an “app end” p ermission, by sending new data. The p oisoning attack against online centroid anomaly detection has b een considered by Nelson and Joseph (2006) for the case of inﬁnite window, i.e., when a learning algorithm memorizes all data seen so far. Their main result w as surprisingly optimistic: it w as shown that the num b er of attac k data p oin ts to b e injected grows exp onen tially as a function of the impact ov er a learned hypothesis. Ho wev er, the assumption of an inﬁnite window also hinders the abilit y of a learning algorithm to adjust to legitimate c hanges in the data distribution. As a main contribution of this work, we present the security analysis of online centroid anomaly detection for the ﬁnite window case, i.e., when only a ﬁxed num b er of data p oints can be used at an y time to form a h yp othesis. W e show that, in this case, an attack er 3 can easily compromise a learning algorithm b y using only a linear amount of injected data unless additional constraints are imp osed. As a further con tribution, we analyze the algo- rithm under tw o additional constrain ts on the attac ker’s part: (a) the fraction of the traﬃc con trolled by an attack er is b ounded by ν , and (b) the false p ositiv e rate induced by an at- tac k is b ounded by α . Both of such constrain ts can b e motiv ated b y an op erational practice of anomaly detection systems. Overall, w e signiﬁcantly extend the analysis of Nelson and Joseph (2006) b y considering a more realistic learning scenario, explicitly treating p otential constrain ts on the attac ker’s part and pro viding tighter b ounds. The metho dology of our analysis follo ws the following framew ork, which we b elieve can b e used for a quantitative se curity analysis of learning algorithms (Lasko v and Kloft, 2009): 1. Axiomatic formalization of the le arning and attack pr o c esses . The ﬁrst step in the analysis is to formally sp ecify the learning and attack processes. Suc h formalization includes deﬁnitions of data sources and ob jectiv e (risk) functions used b y each part y , as well as the attack goal. It sp eciﬁes the kno wledge a v ailable to an attac ker, i.e., whether he knows an algorithm, its parameters and in ternal state, and which data he can p oten tially manipulate. 2. Sp e ciﬁc ation of an attacker’s c onstr aints . Poten tial constrain ts on the attack er’s part ma y include: percentage of traﬃc under his control, amoun t of additional data to b e injected, an upp er b ound on the norm of manipulated part, a maximal allow able false-p ositiv e rate (in case an attack must stealthy), etc. Such constraints m ust b e incorp orated in to the axiomatic formalization. 3. Investigation of an optimal attack p olicy . Given a formal description of the problem and constrain ts, an optimal attack p olicy must b e in vestigated. Suc h p olicy may b e long-term, i.e., o ver m ultiple attack iteration, as well as short-term, for a single iteration. In vestigation can b e carried out either as a formal pro of or numerically , b y casting the search for an attac k p olicy as an optimization problem. 4. Bounding of an attacker’s gain under an optimal p olicy . The ultimate goal of our analysis is to quantify an attack er’s gain or eﬀort under his optimal p olicy . Suc h analysis ma y take diﬀerent forms, for example calculation of the probability for an attac k to succeed, estimation of the required num b er of attack iterations, calculation of the geometric impact of an attack (a shift to w ards an insecure state), etc. Organization of this pap er reﬂects the main steps of the prop osed metho dology . In a preliminary Section 2 the mo dels of the learning and the attack pro cesses are introduced. The analytical part is arranged in t wo sections as follo ws. Section 4 addresses the steps (1), (3) and (4) under an assumption that an attack er has full control of the netw ork traﬃc. Section 5 in tro duces an additional assumption that attack er’s con trol is limited to a certain ﬁxed fraction of net work traﬃc, as required in step (2). Another constraint of the b ounded false p ositiv e rate is considered in Section 6. This section also remo v es a somewhat unrealistic assumption of Section 5 that all inno cuous p oints are accepted b y the algorithm. The analytic results are exp erimen tally veriﬁed in Section 7 on real HTTP data and attac ks used in intrusion detection systems. Some pro ofs and the auxiliary technical material are presen ted in the App endix. 4 Before moving on to the detailed presentation of our analysis, it may b e instructiv e to discuss the place of a p oisoning attac k in the ov erall attack taxonom y and practical implication of its assumptions. F or t wo-class learning problems, attacks against learning algorithms can b e generally classiﬁed according to the following tw o criteria (the terminology in the taxonomy of Barreno et al. (2006) is giv en in brack ets): • whether an attack is staged during the training (causative) or the deplo yment of an algorithm (causative/exploratory), or • whether an attack attempts to increase the false negativ e or the false p ositive rate at the deploymen t stage (in tegrity/a v ailability). The p oisoning attac k addressed in our work can b e classiﬁed as a causativ e integrit y attac k. This scenario is quite natural, e.g., in w eb application scenarios in which the data on a server can b e assumed secure but the injection of adv ersarial data cannot b e easily prev ented. Other common attack t yp es are a mimicry attack – alteration of malicious data to resemble inno cuous data (an exploratory in tegrity attac k), or a “red herring” attack – sending of junk data that causes false alarms (an exploratory av ailabilit y attack). Attac ks of the latter t wo kinds are b ey ond the scop e of our in vestigation. As a ﬁnal remark, we m ust consider the extent to whic h the attac ker is familiar with the learning algorithm and trained model. One of the k ey principles of computer security , kno wn as Ker ckhoﬀ ’s principle , is that the robustness of any security instrument must not dep end on k eeping its op erational functionality secret. Similar to mo dern cryptographic metho ds, w e must assume that the attack er knows which mac hine learning algorithm is deploy ed and ho w it operates (he can ev en use machine learning to rev erse engineer deploy ed classiﬁers, as shown by Lo wd and Meek (2005b)). A more serious diﬃculty on the attac ker’s part may b e to get hold of the training data or of the particular learned mo del. In the case of anomaly detection, it is relatively easy for an attac ker to retriev e a learned mo del: it suﬃc es to sniﬀ on the same application that is protected by an algorithm to get appro ximately the same inno cuous data the algorithm is trained on. Hence, we will assume that an attac ker has precise knowledge of the trained mo del at any time during the attac k. 2. Learning and A ttac k Mo dels Before pro ceeding with the analysis, we ﬁrst presen t the precise mo dels of the learning and the attack pro cesses. Our fo cus on anomaly detection is motiv ated by its abilit y to detect p oten tially nov el attacks, a crucial demand of mo dern information security . 2.1 Centroid Anomaly Detection Giv en the data set X = { x 1 , . . . , x n } , the goal of anomaly detection (also often referred to as “nov elt y detection”) is to determine whether an example x is unlik ely to hav e b een generated by the same distribution as the set X . A natural wa y to p erform anomaly detection is to estimate a probabilit y densit y function of the distribution from which the set X w as drawn and ﬂag x as anomalous if it comes from a region with low density . In general, how ev er, density estimation is a diﬃcult problem, espec ially in high dimensions. A large amount of data is usually needed to reliably estimate the density in all regions of 5 Figure 1: Illustration of the density lev el estimation using a cen troid mo del with a non- linear kernel. the space. F or anomaly detection, kno wing the density in the entire space is sup erﬂuous, as w e are only interested in deciding whether a sp eciﬁc p oint falls into a “sparsely p opulated” area. Hence several direct metho ds hav e b een prop osed for anomaly detection, e.g., one- class SVM (Sc h¨ olk opf et al., 2001), supp ort v ector data description (SVDD) (T ax and Duin, 1999a,b), and density level set estimation (Polonik, 1995; Tsybak ov, 1997; Steinw art et al., 2005). A comprehensiv e survey of anomaly detection techniques can b e found in Markou and Singh (2003a,b). In the cen troid anomaly detection, a Euclidean distance from an empirical mean of the data is used as a measure of anomality: f ( x ) = || x − 1 n n X i =1 x i || . If a hard decision is desired instead of a soft anomalit y score, the data p oin t is considered anomalous if its anomaly score exceeds a ﬁxed threshold r . Cen troid anomaly detection can b e seen as a sp ecial case for the SVDD with outlier fraction η = 1 and of the P arzen window density estimator (P arzen, 1962) with the Gaussian k ernel function k ( x , y ) = 1 √ 2 π exp( − 1 2 x · y ). Despite its straigh tforwardness, a centroid mo del can represent arbitrary complex density level sets using a kernel mapping (Sch¨ olk opf and Smola, 2002; M ¨ uller et al., 2001) (see Fig. 1). It has b een successfully used in a v ariety of anomaly detection applications such as in trusion detection (Hofmeyr et al., 1998; Y eung and Chow, 2002; Lask ov et al., 2004a; W ang and Stolfo, 2004; Rieck and Lasko v, 2006; W ang et al., 2006; Rieck and Lasko v, 2007), wireless sensor net works (Ra jasegarar et al., 2007) and jet engine vibration data analysis (Nairac et al., 1999). It can b e shown (cf. Sha we-T a ylor and Cristianini (2004), Section 4.1) that even in high-dimensional spaces induced by nonlinear feature maps, the empirical estimator of the center of mass of the data is stable and the radius of a sphere 6 anc hored at the cen ter of mass is related to a level set of the corresp onding probability densit y . 2.2 Online Anomaly Detection The ma jority of anomaly detection applications ha ve to deal with non-stationary data. This is esp ecially typical for computer security , as usually the pro cesses b eing monitored change o ver time: e.g., net work traﬃc proﬁle is strongly inﬂuenced b y the time of the day and system call sequences dep end on the applications running on a computer. Hence the mo del of normalit y constructed by anomaly detection algorithms usually needs to b e up dated during their op erations. In the extreme case, such an up date can b e p erformed after the arriv al of eac h data p oin t resulting in the online operation. Ob viously , re-training the mo del from scratch every time is computationally infeasible; how ever, incorp oration of new data p oin ts and the remov al of irrelev ant ones can b e done with acceptable eﬀort (Lasko v et al., 2006). F or the centroid anomaly detection, re-calculation of the cen ter of mass is straightfor- w ard and requires O (1) work. If all examples are “memorized”, i.e., the index n is gro wing with the arriv al of eac h example, the index n is incremented for ev ery new data p oint, and the up date is computed as 2 c 0 =  1 − 1 n  c + 1 n x . (1) F or the ﬁnite horizon, i.e. constant n , some previous example x i is replaced by a new one, and the up date is p erformed as c 0 = c + 1 n ( x − x i ) . (2) V arious strategies can b e used to determine the “least relev ant” p oint x i to b e remov ed from a working set: (a) oldest-out : The p oint with the oldest timestamp is remov ed. (b) random-out : A randomly c hosen p oint is remov ed. (c) nearest-out : The nearest-neighbor of the new p oint x is remo ved. (d) average-out : The cen ter of mass is remov ed. The new cen ter of mass is recalculated as c 0 = c + 1 n ( x − c ), whic h is equiv alent to Eq. (1) with constant n . The strategies (a)–(c) require the storage of all p oints in the working set, whereas the strategy (d) can b e implemented b y holding only the center of mass in memory . 2.3 Poisoning attac k The goal of a p oisoning attac k is to force an anomaly detection algorithm to accept an attac k p oin t A that lies outside of the normal ball, i.e., || A − c || > r . It is assumed that 2. The update formula can b e generalized to c 0 = c + κ n ( x − x i ), with ﬁxed κ ≥ 1. The b ounds in the analysis c hange only by a constan t factor, which is negligible. 7 A Attack Locations Before Attack After Attack Center Figure 2: Illustration of a p oisoning attack. By iteratively inserting malicious training p oin ts an attack er can gradually corrupt “drag” the centroid in to a direction of an attack. an attack er kno ws the anomaly detection algorithm and all the training data. Ho wev er, an attac ker cannot mo dify any existing data except for adding new p oints. These assumptions mo del a scenario in whic h an attack er can sniﬀ data on the wa y to a particular host and can send his own data, while not ha ving write access to that host. As illustrated in Fig. 2, the p oisoning attack attempts to inject sp ecially crafted p oin ts that are accepted as inno cuous and push the cen ter of mass in the direction of an attac k p oin t un til the latter app ears inno cuous. What points should be used by an attac ker in order to subv ert online anomaly detection? In tuitively one can exp ect that the optimal one-step displacement of the center of mass is ac hieved b y placing attac k p oin t x i at the line connecting c and A such that || x i − c || = r . A formal pro of of the optimality of such strategy and estimation of its eﬃciency constitutes the main ob jectiv e of securit y analysis of online anomaly detection. In order to quantify the eﬀectiv eness of a p oisoning attac k, we deﬁne the i -th relative displacemen t of the center of mass. This quantit y measures the relative length of the pro jection of c i on to the “attac k direction” a in terms of the radius of the normalit y ball. Deﬁnition 1 (Relativ e displacement) (a) L et A b e an attack p oint and deﬁne by a = A − c 0 || A − c 0 || the ac c or ding attack dir e ction ve ctor. The i -th relative displacement , denote d by D i , is deﬁne d as D i = ( c i − c 0 ) · a r . W.l.o.g. we assume that c 0 = 0 . (b) Attack str ate gies maximizing the displac ement D i in e ach iter ation i ar e r eferr e d to as greedy optimal attack strategies . 8 3. A ttac k Eﬀectiveness for Inﬁnite Horizon Centroid Learner The eﬀectiveness of a p oisoning attack for an inﬁnite horizon has b een analyzed in Nelson and Joseph (2006). W e provide an alternative proof that follows the framework prop osed in the introduction. Theorem 2 The i -th r elative displac ement D i of the online c entr oid le arner with an inﬁnite horizon under the p oisoning attack is b ounde d by D i ≤ ln  1 + i n  , (3) wher e i is the numb er of attack p oints and n the numb er of initial tr aining p oints. Pro of W e ﬁrst determine an optimal attack strategy and then b ound the attac k progress. (a) Let A b e an attack point and denote b y a the corresponding attac k direction v ector. Let { a i | i ∈ N } b e adversarial training p oints. The center of mass at the i -the iteration is giv en in the follo wing recursion: c i +1 =  1 − 1 n + i  c i + 1 n + i a i +1 , (4) with initial v alue c 0 = 0. By the construction of the p oisoning attack, || a i − c i || ≤ r , which is equiv alent to a i = c i + b i with || b i || ≤ r . Eq. (4) can thus b e transformed into c i +1 = c i + 1 n + i b i . T aking scalar pro duct with a and using the deﬁnition of a relative displacement, w e obtain: D i +1 = D i + 1 n + i · b i · a r , (5) with D 0 = 0. The right-hand side of the Eq. (5) is clearly maximized under || b i || ≤ 1 by setting b i = r a . Th us the optimal attac k is deﬁned b y a i = c i + r a . (6) (b) Plugging the optimal strategy b i = r a into Eq (5), w e hav e: D i +1 = D i + 1 n + i . This recursion can b e explicitly solv ed, taking into accoun t that d 0 = 0, resulting in: D i = i X k =1 1 n + k = n + i X k =1 1 k − n X k =1 1 k . Inserting the upp er b ound on the harmonic series, P m k =1 1 k = ln( m ) +  m with  m ≥ 0 in to the ab o ve formula, and noting that  m is monotonically decreasing, we obtain D i ≤ ln( n + i ) − ln( n ) = ln  n + i n  = ln  1 + i n  , 9 whic h completes the pro of. Since the b ound in Eq. (3) is monotonically increasing, w e can inv ert it to obtain the estimate of the eﬀort needed b y an attac k er to ac hieve his goal: i ≥ n · (exp ( D ∗ ) − 1) . It can b e seen that an eﬀort need to p oison a online cen troid learner is exp onential in terms of the relativ e displacement of the center of mass. 3 In other w ords, an attac ker’s eﬀort gro ws prohibitively fast with resp ect to the separabilit y of an attac k from the inno cuous data. Ho wev er, this is not surprising since due the inﬁnitely gro wing training window the con tribution of new p oin ts to the computation of the center of mass is steadily decreasing. 4. P oisoning A ttack against Finite Horizon Centroid Learner As it w as shown in Section 2.3, the p oisoning attack is ineﬀective against online centroid anomaly detection if all p oints are kept “in memory”. Unfortunately , memorizing the p oin ts defeats the main purp ose of online algorithms, i.e., their abilit y to adjust to non- stationarit y 4 . Hence it is imp ortant to understand how the remov al of data p oin ts from a w orking set aﬀects the security of online anomaly detection. F or that, the sp eciﬁc remov al strategies presented in Section 2.2 m ust b e considered. It will turn out that for the a verage- and random-out rules the analysis can b e carried out theoretically . F or the nearest-out rule the analysis is more complicated but an optimal attac k can b e stated as mathematical optimization problem, and the attack eﬀectiv eness can b e analyzed empirically . 4.1 Poisoning A ttack for Av erage- and Random-out Rules W e b egin our analysis with the a verage-out learner which follo ws exactly the same up date rule as the inﬁnite-horizon online centroid learner with the exception that the windo w size n remains ﬁxed instead of gro wing indeﬁnitely (cf. Section 2.2). Despite the similarity to the inﬁnite-horizon case, the result presented in the follo wing theorem is surprisingly p essimistic. Theorem 3 The i -th r elative displac ement D i of the online c entr oid le arner with the aver age- out up date rule under an worst-c ase optimal p oisoning attack is D i = i n , (7) wher e i is the numb er of attack p oints and n is the tr aining window size. Pro of The pro of is similar to the pro of of Theorem 2. By explicitly writing out the recurrence b et ween subsequent displacements, we conclude that the optimal attac k is also 3. Even constraining a maxim um num b er of online up date steps cannot remo ve the b ound’s exp onential gro wth (Nelson and Joseph, 2006). 4. Once again we remark that the data need not b e physically stored, hence the memory consumption is not the main b ottleneck in this case. 10 attained b y placing an attac k p oin t on the line connecting c i and a at the edge of the sphere (cf. Eq. (6)): a i = c i + r a . It follows that the relativ e displacement under the optimal attac k is D i +1 = D i + 1 n . Since this recurrence is indep endent of the running index i , the displacemen t is simply ac- cum ulated ov er each iteration, which yields the b ound of the theorem. One can see, that unlik e the logarithmic b ound in Theorem 2, the av erage-out learner is characterized by a linear b ound on the displacement. As a result, an attac ker only needs a linear amount of injected p oin ts – instead of an exp onential one – in order to sub v ert an a verage-out learner. This cannot b e considered secure. W e obtain a similar result for the random-out remo v al strategy . Theorem 4 F or the i -th r elative displac ement D i of the online c entr oid le arner with the r andom-out up date rule under an worst-c ase optimal p oisoning attack it holds E ( D i ) = i n , (8) wher e i is the numb er of attack p oints, n is the tr aining window size, and the exp e ctation is dr awn over the choic e of the r emove d data p oints. Pro of The pro of is based on the observ ation that the random-out rule in exp ectation b oils do wn to a verage-out, and hence is reminiscen t to the pro of of Th. 3. 4.2 Poisoning A ttack for Nearest-out Rule Let us consider the alternativ e up date strategies mentioned in Section 2.1. The up date rule c 0 = c + 1 n ( x − x 0 ) of the oldest-out strategy is essentially equiv alent to the up date rule of the a verage-out except that the outgoing cen ter c is replaced by the oldest p oin t x 0 . In b oth cases the p oint to b e remov ed is ﬁxed in adv ance regardless of an attack er’s mov es, hence the p essimistic result dev eloped in Section 4.1 remains v alid for this case. On a verage, the random-out up date strategy is – despite its nondeterministic nature – equiv alen t to the a verage-out strategy . Hence, it also cannot b e considered secure against a p oisoning attac k. One might exp ect that the nearest-out strategy p oses a stronger challenge to an attack er, as it tries to k eep as m uch of a w orking set div ersit y as possible b y retaining the most similar data to a new p oint. It turns out, how ev er, that ev en this strategy can b e broken with a feasible amoun t of wor k if an attack er follows a greedy optimal strategy . The latter is a sub ject of our in vestigation in this section. 11 4.2.1 An optimal a tt ack Our in vestigation fo cuses on a gr e e dy optimal attac k, i.e., an attac k that pro vides a maximal gain for an attack er in a single iteration. F or the inﬁnite-horizon learner (and hence also for the a v erage-out learner, as it uses the same recurrence in a pro of ), it is possible to sho w that the optimal attac k yields the maximum gain for the entire sequence of attack iterations. F or the nearest-out learner, it is hard to analyze a full sequence of attac k iterations, hence w e limit our analysis to a single-iteration gain. Empirically , even a greedy optimal attack turns out to b e eﬀective. T o construct a greedy optimal attack, it suﬃces to determine for eac h p oin t x i the lo cation of an optimal attack point x ∗ i to replace x i . This can be form ulated as the follo wing optimization problem: Optimization Problem 5 (greedy optimal attac k) { x ∗ i , f i } = max x ( x − x i ) · a (9.a) s.t. k x − x i k ≤ k x − x j k , ∀ j = 1 , ..., n (9.b) k x − 1 n P n j =1 x j k ≤ r . (9.c) The ob jectiv e of the optimization problem 5 reﬂects an attack er’s goal of maximizing the pro jection of x − x i on to the attack direction vector a . The constraint (9.b) sp eciﬁes the condition that the p oint x i is the nearest neighbor of x (i.e., x falls in to a V or onoi c el l induced b y x i ). The constrain t (9.c), when active, enforces that no solution lies outside of the sphere. Hence the geometric in tuition b ehind an optimal attac k, illustrated in Figure 3, is to replace some p oin t with an attack p oint placed at the “corner” of the former’s V oronoi cell (including p ossibly a round b oundary of the centroid) that provides a highest displaceme n t of the center in the attac k p oint’s direction. The maximization of Eq. (9) ov er all p oints in a current working set yields the index of the p oin t to b e replaced by an attack er: α = argmax i ∈ 1 ,...,n f i (10) By plugging the deﬁnition of a Euclidean norm into the inner optimization problem (9) and multiplyin g out the quadratic constrain ts, all but one norm constrain ts reduce to simpler linear constraints: { x ∗ i , f i } = max x ( x − x i ) · a (11.a) s.t. 2( x j − x i ) · x ≤ x j · x j − x i · x i , ∀ j = 1 , ..., n (11.b) x · x − 2 n P n j =1 x · x j ≤ r 2 − 1 n 2 P n j,k =1 x j · x k . (11.c) Due to the quadratic constrain t (11.c), the inner optimization task is not as simple as a linear or a quadratic program. How ev er, several standard optimization pack ages, e.g., CPLEX or MOSEK, can handle such so-called quadratically constrained linear programs (QCLP) rather eﬃciently , esp ecially when there is only one quadratic constraint. Alternatively , one can use sp ecialized algorithms for linear programming with a single quadratic constrain t (v an de P anne, 1966; Martein and Schaible, 2005) or conv ert the quadratic constrain t to a second-order cone (SOC) constraint and use general-purp ose conic optimization methods. 12 A x x i training data A = target x = attack location x i = replaced point center before the attack Attack Direction Figure 3: The geometry of a p oisoning attac k for the nearest-out rule. An optimal attack is achiev ed at the b oundary of a V oronoi cell. 4.2.2 Implement a tion of a greedy optimal a tt a ck Some additional work is needed for a practical implementation of a greedy optimal attack against a nearest-out learner. A p oin t can b ecome “immune” to a p oisoning attack, if its V oronoi cell do es not ov erlap with the hypersphere of radius r centered at c k , at some iteration k . The quadratic con- strain t (9.c) is never satisﬁed in this case, and the inner optimization problem (9) b ecomes infeasible. F rom then on, a p oin t remains in the working set forev er and slo ws do wn the attac k progress. T o a v oid this a wkward situation, an attack er m ust k eep trac k of all optimal solutions x ∗ i of the inner optimization problems. If any x ∗ i slips out of the hypersphere after replacing the p oint x α with x ∗ α , an attack er should ignore the outer lo op decision (10) and instead replace x i with x ∗ i . A signiﬁcant sp eedup can b e attained by av oiding the solution of unnecessary QCLP problems. Let S = { 1 , . . . , i − 1 } and α S b e the current b est solution of the outer lo op problem (10) o ver the set S . Let f α S b e the corresponding ob jectiv e v alue of an inner optimization problem (11). Consider the follo wing auxiliary quadratic program (QP): max x k x − 1 n P n j =1 x j k (12.a) s.t. 2( x j − x i ) · x ≤ x j · x j − x i · x i , ∀ j = 1 , ..., n (12.b) ( x − x i ) · a ≥ f α S . (12.c) Its feasible set comprises the V oronoi cell of x i , deﬁned by constrain ts (12.b), further reduced b y constraint (12.c) to the p oints that improv e the curren t v alue f α S of the global ob jective function. If the ob jective function v alue pro vided b y the solution of the auxiliary QP (12) exceeds r then the solution of the lo cal QCLP (11) do es not provide an improv ement of the global ob jectiv e function f α S . Hence an exp ensive QCLP optimization can b e skipp ed. 13 4.2.3 A tt ack Effectiveness T o ev aluate the eﬀectiveness of a greedy optimal attac k, we p erform a simulation on an artiﬁcial geometric data. The goal of this sim ulation is in vestigate the b ehavior of the relativ e displacement D i during the progress of a greedy optimal attack. An initial working set of size n = 100 is sampled from a d -dimensional Gaussian distri- bution with unit cov ariance (exp erimen ts are repeated for v arious v alues of d ∈ { 2 , ..., 100 } ). The radius r of the online centroid learner is chosen suc h that the exp ected false p ositive rate is b ounded by α = 0 . 001. An attac k direction a , k a k = 1 is c hosen randomly , and 500 attac k iterations (5 ∗ n ) are generated using the pro cedure presen ted in Sections 4.2.1 – 4.2.2. The relative displacement of the center in the direction of attack is measured at each iteration. F or statistical signiﬁcance, the results are a veraged ov er 10 runs. Figure 4(b) shows the observed progress of the greedy optimal attack against the nearest- out learner and compares it to the b eha vior of the theoretical b ounds for the inﬁnite-horizon learner (the b ound of Nelson et al.) and the av erage-out learner. The attack eﬀectiveness is measured for all three cases by the relativ e displacemen t as a function of the n umber of iterations. Plots for the nearest-out learner are presented for v arious dimensions d of the artiﬁcial problems tested in simulations. The following tw o observ ations can b e made from the plots provided in Figure 4(a): Firstly , the attack progress, i.e., the functional dep endence of the relativ e displacement of the greedy optimal attack against the nearest-out learner with respect to the n umber of iterations, is line ar . Hence, con trary to the initial intuition, the remo v al of nearest neigh b ors to incoming p oints do es not add securit y against a p oisoning attac k. Secondly , the slop e of the linear attack progress incr e ases with the dimensionality of the pr oblem . F or low dimensionality , the relative displacement of the nearest-out learner is comparable, in absolute terms, with that of the inﬁnite-horizon learner. F or high di- mensionalit y , the nearest-out learner b ecomes even less secure than the simple a v erage-out learner. By increasing the dimensionality b eyond d > n the attack eﬀectiveness cannot b e increased. Mathematical reasons for suc h b ehavior are inv estigated in Section B.1. A further illustration of the behavior of the greedy optimal attack is giv en in Figure 4(b), sho wing the dep endence of the av erage attack slop e on the dimensionality . One can see that the attack slop e increases logarithmically with the dimensionalit y and wanes out to a constan t factor after the dimensionality exceeds the num b er of training data p oints. A theoretical explanation of the observ ed exp erimental results is giv en in the next section. 4.3 Concluding Remarks T o summarize our analysis for the case of attack er’s full con trol ov e r the data, we conclude that an optimal p oisoning attac k can successfully subv ert a ﬁnite-horizon online cen troid learner for all outgoing p oin t selection rules. This conclusion contrasts with the analysis of the inﬁnite-horizon learner carried out in Barreno et al. (2006) that yields a logarithmic attac k progress. As a compromise, one can in practice choose a large working set size n , whic h reduces the slop e of a linear attack progress. Among the diﬀeren t outgoing p oint selection rules, the nearest-out rule presents some c hallenges to the implemen tation of an optimal attac k; ho wev er, some appro ximations can mak e such an attack feasible while still main taining a reasonable progress rate. The k ey 14 0 1 2 3 4 5 0 1 2 3 4 5 6 i/n D average−out (finite horizon) average−out (infinite horizon) nearest−out d=16 d=8 d=4 d=100 d=2 (a) 10 1 10 2 0.2 0.4 0.6 0.8 1 1.2 dimensionality slope (b) Figure 4: Eﬀectiveness of a p oisoning attack for the nearest-out rule as a function of input space dimensionalit y . The displacement of a cen troid in to a direction of an attack gro ws linearly with the num ber of injected p oints. The slop e of the linear gro wth increases with the input space dimensionalit y . Upp er b ounds on the displacemen t of the av erage-out rule rule are plotted for comparison. factor for the success of a p oisoning attack in the nearest-out case lies in the high dimen- sionalit y of the feature space. The progress of an optimal p oisoning attac k dep ends on the size of V oronoi cells induced b y the training data p oints. The size of V oronoi cells is related linearly to the v olume of the sphere corresp onding to attac k’s feasible region. The increasing dimensionality of a feature space blows up the volume of the sphere and hence causes a higher attack progress rate. In the following sections w e analyze tw o additional factors that can aﬀect the progress of a poisoning attac k. First, w e consider the case of an attac ker b eing able to con trol only a ﬁxed fraction ν of the training data. Subsequen tly we analyze a scenario in which an attac ker is not allo wed to exceed a certain false p ositive rate α , e.g., b y stopping online learning when a high false p ositive rate is observ ed. In will b e shown that b oth of these p ossible constrain ts signiﬁcan tly reduce the eﬀectiveness of a p oisoning attack. 5. P oisoning A ttack with Limited Bandwidth Constraint W e now pro ceed with inv estigation of a p oisoning attack under a limited bandwidth con- strain t imp osed on an attac ker. W e assume that an attack er can only inject up to a fraction of ν of the training data. In security applications, such an assumption is natural, as it ma y b e diﬃcult for an attack er to surpass a certain amoun t of inno cuous traﬃc. F or simplicit y , w e restrict ourselv es to the av erage-out learner, as we hav e seen that it only diﬀers b y a constan t from a nearest-out one and in expectation equals a random-out one. 15 5.1 Learning and Attac k mo del The initial online centroid learner is centered at the p osition X 0 and has the radius r (w.l.o.g. assume X 0 = 0 and r = 1). A t each iteration a new training p oint arrives which is either inserted by an adv ersary or is drawn indep endently from the distribution of innocuous p oin ts, and a new cen ter of mass X i is calculated 5 . The mixing of inno cuous and attack p oin ts is mo deled by a Bernoulli random v ariable with the parameter ν . Adversarial p oin ts A i are c hosen according to an attac k function f dep ending on the actual state of the learner X i . The inno cuous p o ol is mo deled by a probabilit y distribution, from which the innocuous p oin ts  i are indep endently dra wn. W e assume that the exp ectation of inno cuous p oin ts  i coincides with the initial center of mass: E (  i ) = X 0 . F urthermore, we assume that all inno cuous p oin ts are accepted b y the initial learner, i.e., k  i − X 0 k ≤ r . Moreo ver, for didactical reasons, we mak e a rather artiﬁcial assumption, which we will drop in the next chapter: al l inno cuous p oints ar e ac c epte d by the le arner, at any time of the attack, indep endent of their actual distanc e to the c enter of mass . In the next section w e drop this assumption, suc h that the learner only accept p oin ts which fall within the actual radius. The describ ed probabilistic mo del is formalized b y the follo wing axiom. Axiom 6 { B i | i ∈ N } ar e indep endent Bernoul li r andom variables with p ar ameter ν > 0 .  i ar e i.i.d. r andom variables in a r epr o ducing kernel Hilb ert sp ac e H , dr awn fr om a ﬁxe d but unknown distribution P  , satisfying E (  i ) = 0 and k  i k ≤ r = 1 for e ach i . B i and  j ar e mutual ly indep endent for e ach i, j . f : H → H is an attack str ate gy satisfying k f ( x ) − x k ≤ r . { X i | i ∈ N } is a c ol le ction of r andom ve ctors such that X 0 = 0 and X i +1 = X i + 1 n ( B i f ( X i ) + (1 − B i )  i − X i ) . (13) F or simplicit y of notation, we in this section refer to a collection of random v ectors { X i | i ∈ N } satisfying Axiom 6 as online c entr oid le arner denoted b y C . F urthermore w e denote  :=  · a . Any function f satisfying Ax. 6 is called attack str ate gy . According to the abov e axiom an adv ersary’s attack strategy is formalized by an arbi- tr ary function f . This raises the question whic h attack strategies are optimal in the sense that an attac ker reaches his goal of concealing a predeﬁned attack direction v ector in a minimal n umber of iterations. An attac k’s progress is measured b y pro jecting the curren t cen ter of mass on to the attac k direction vector: Deﬁnition 7 (a) L et a b e an attack dir e ction ve ctor (w.l.o.g. || a || = 1 ), and let C = { X i | i ∈ N } b e a online c entr oid le arner. The i -th displacement of C , denote d by D i , is deﬁne d by D i = X i · a R . (b) Attack str ate gies maximizing the displac ement D i in e ach iter ation i ar e r eferr e d to as optimal attack strategies . 5. T o emphasize the probabilistic mo del used in this section, we denote the lo cation of a center and the relativ e displacement by capital letters. 16 5.2 An Optimal Attac k The following result characterizes an optimal attack strategy for the mo del sp eciﬁed in Axiom 6. Prop osition 8 L et a b e an attack dir e ction ve ctor and let C b e a c entr oid le arner. Then the optimal attac k strategy f is given by f ( X i ) := X i + a . (14) Pro of Since by Axiom 6 we hav e k f ( x ) − x k ≤ r , an y v alid attack strategy can b e written as f ( x ) = x + g ( x ), such that k g k ≤ r = 1 . It follows that D i +1 ≤ X i +1 · a =  X i + 1 n ( B i f ( X i ) + (1 − B i )  i − X i )  · a = D i + 1 n ( B i D i + B i g ( X i ) · a + (1 − B i )  i − D i ) . Since B i ≥ 0, the optimal attac k strategy should maximize g ( X i ) · a sub ject to || g ( X i ) || ≤ 1. The maximum is clearly attained b y setting g ( X i ) = a . 5.3 Attac k Eﬀectiv eness The estimate of an optimal attack’s eﬀectiveness in the limited con trol case is giv en in the follo wing theorem. Theorem 9 L et C b e a c entr oid le arner under an optimal p oisoning attack. Then, for the displac ement D i of C , it holds: (a) E ( D i ) = (1 − c i ) ν 1 − ν (b) V ar( D i ) ≤ γ i  ν 1 − ν  2 + δ n wher e γ i = c i − d i , c i :=  1 − 1 − ν n  i , d i =  1 − 1 − ν n  2 − 1 n  i and δ n := ν 2 +(1 − d i ) (2 n − 1)(1 − ν ) 2 . Pro of (a) Inserting the optimal attac k strategy of Eq. (14) into Eq. (13) of Ax. 6, we ha v e: X i +1 = X i + 1 n ( B i ( X i + a ) + (1 − B i )  i − X i ) , whic h can be rewritten as: X i +1 =  1 − 1 − B i n  X i + B i n a + (1 − B i ) n  i . (15) 17 0 0.5 1 1.5 2 0 1 2 3 4 5 i/n D i average−out (finite horizon) average−out (infinite horizon) ν =5% ν =100% ν =50% ν =20% Figure 5: Theoretical b ehavior of the displacemen t of a centr oid under a p oisoning attac k for a b ounded fraction of traﬃc under attack er’s con trol. The inﬁnite horizon b ound of Nelson et al. is sho wn for comparison (solid line). T aking the exp ectation on the latter equation, and noting that by Axiom 6 E (  ) = 0 and E ( B i ) = ν holds, we ha ve E ( X i +1 ) =  1 − 1 − ν n  E ( X i ) + ν n a , whic h by Def. 7 translates to E ( D i +1 ) =  1 − 1 − ν n  E ( D i ) + ν n . The statement (a) follows from the latter recursive eequation b y Prop. 17 (formula of the geometric series). F or the more demanding proof of (b), see App endix B.2. The following corollary shows the asymptotic b ehavior of the abov e theorem. Corollary 10 L et C b e a c entr oid le arner satisfying under an optimal p oisoning attack. Then. for the displac ement D i of C , it holds: (a) E ( D i ) ≤ ν 1 − ν for al l i (b) V ar( D i ) → 0 for i, n → ∞ . Pro of The corollary follo ws b y γ i , δ n → 0 for i, n → ∞ . The gro wth of the ab o ve b ounds as a function of an num b er of attack iterations is illustrated in Fig. 5.3. One can see that the attack’s success strongly dep ends on the fraction of the training data controlled b y an attac ker. F or small ν , the attac k progress is b ounde d by a c onstant , which implies that an attack fails even with an inﬁnite eﬀort. 18 0 1 2 3 4 5 0 0.01 0.02 0.03 0.04 0.05 0.06 i/n D i empirical displacement theoretical displacement Figure 6: Comparison of empirical displacemend of the cen troid under p oisoning attac k with attac k er’s limited con trol ( ν = 0 . 05) with a theoretical b ound for the same setup. Emprical results are a veraged o ver 10 runs; standard deviation is shown b y vertical bars. This result provides a much str onger se curity guar ante e than the exp onential b ound for the inﬁnite horizon case. T o empirically inv estigate the tigh tness of the deriv ed bound we compute a Mon te Carlo sim ulation of Axiom 6 with the parameters ν = 0 . 05, n = 100000, H = R 2 , and  b eing a uniform distribution ov er the unit circle. Fig. 5.3 shows a t ypical displacement curve o ver the ﬁrst 500 , 000 attac k iterations. Errorbars are computed ov er 10 rep etitions of the sim ulation. 6. P oisoning A ttack under F alse Positiv e Constraints In the last section w e hav e assumed, that inno cuous training p oints  i are alw ays accepted b y the online centroid learner. But while an attack er displaces the h yp ersphere, it may happ en that some inno cuous p oin ts drop out of the hypersphere’s b oundary . W e hav e seen that an attac ker’s impact highly dep ends on the fraction of p oints he places. If an attack er succeeds in pushing the hypersphere far enough suc h that suﬃcien tly man y innocuous p oin ts drop out, he can quic kly displace the h yp ersphere. 6.1 Learning and Attac k Mo del Motiv ated by the ab o ve considerations w e mo dify the probabilistic mo del of the last section as follo ws. Again we consider a online centroid learner initially anchored at a p osition X 0 ha ving a radius r , for the sake of simplicity and without loss of generalit y X 0 = 0 and r = 1. Then inno cuous and adversarial p oints are mixed into the training data according to a ﬁxed fraction, controlled by a binary v alued random v ariable B i . But now, in contrast to the last section, inno cuous p oints  i are only accepted if and only if they fall within a 19 radius of r of the h yp ersphere’s center X i . In addition, to av oid the learner b eing quic kly displaced, we require that the false alarm rate is b ounded by α . If the latter is exceeded, w e assume the adversary’s attac k to hav e failed, i.e., a safe state of the learner is loaded and the online up date mechanism is temp orarily switched oﬀ. W e formalize the probabilistic mo del as follows: Axiom 11 { B i | i ∈ N } ar e indep endent Bernoul li r andom variables with p ar ameter ν > 0 .  i ar e i.i.d. r andom variables in a r epr o ducing kernel Hilb ert sp ac e H , dr awn fr om a ﬁxe d but unknown distribution P  = P −  , satisfying E (  i ) = 0 , and k  i k ≤ r = 1 for e ach i . B i and  j ar e mutual ly indep endent for e ach i, j . f : H → H is an attack str ate gy satisfying k f ( x ) − x k ≤ r . { X i | i ∈ N } is a c ol le ction of r andom ve ctors such that X 0 = 0 and X i +1 = X i + 1 n  B i ( f ( X i ) − X i ) + (1 − B i ) I {k  i − X i k≤ r } (  i − X i )  , (16) if E  i  I {k  i − X i k≤ r }  ≤ 1 − α and by X i +1 = 0 elsewise. F or simplicit y of notation, we in this section refer to a collection of random v ectors { X i | i ∈ N } satisfying Ax. 11 as online c entr oid le arner with maximal false p ositive r ate α denoted b y C . An y function f satisfying Ax. 11 is called attack str ate gy . Optimal attack strategies are characterized in term of the displacement as in the previous section (see Def. 7). 6.2 Optimal Attac k and Attac k Eﬀectiveness The following result characterizes an optimal attack strategy for the mo del sp eciﬁed in Axiom 11. Prop osition 12 L et a b e an attack dir e ction ve ctor and let C b e a c entr oid le arner with maximal false p ositive r ate α . Then an optimal attack str ate gy f is given by f ( X i ) := X i + a . Pro of Since b y Axiom 11 w e hav e k f ( x ) − x k ≤ r, any v alid attack strategy can b e written as f ( x ) = x + g ( x ), suc h that k g k ≤ r = 1 . It follo ws that either D i = 0, in whic h case the optimal f is arbitrary , or we hav e D i +1 = X i +1 · a =  X i + 1 n ( B i f ( X i ) + (1 − B i )  i − X i )  · a = D i + 1 n ( B i ( D i + g ( X i )) + (1 − B i )  i − D i ) Since B i ≥ 0, the optimal attac k strategy should maximize g ( X i ) · a sub ject to || g ( X i ) || ≤ 1. The maximum is clearly attained b y setting g ( X i ) = a . The estimate of an optimal attack’s eﬀectiv eness in the limited con trol case is given in the following main theorem of this pap er. 20 Theorem 13 L et C b e a c entr oid le arner with maximal false p ositive r ate α under a p oi- soning attack. Then, for the displac ement D i of C , it holds: (a) E ( D i ) ≤ (1 − c i ) ν + α (1 − ν ) (1 − ν )(1 − α ) (b) V ar( D i ) ≤ γ i ν 2 (1 − α ) 2 (1 − ν ) 2 + ρ ( α ) + δ n wher e c i :=  1 − (1 − ν )(1 − α ) n  i , d i =  1 − 1 − ν n (2 − 1 n )(1 − α )  i , γ i = ( c i − d i ) , ρ ( α ) = α (1 − c i )(1 − d i )(2 ν (1 − α )+ α ) (1 − 1 2 n )(1 − ν ) 2 (1 − α ) 2 , and δ n = (1 − d i )( ν +(1 − ν ) E (  2 i )) (2 n − 1)(1 − ν )(1 − α ) . The pro of is tec hnically demanding and is given in App. B.3. Despite the more general pro of reasoning, we recov er the tightness of the b ounds of the previous section for the sp ecial case of α = 0, as sho wn by the follo wing corollary . Corollary 14 Supp ose a maximal false p ositive r ate of α = 0 . Then, the b ounds on the exp e cte d displac ement D i , as given by Th. 9 and Th. 13, c oincident. F urthermor e, the varianc e b ound of Th. 13 upp er b ounds the one of Th. 9. Pro of W e start b y setting α = 0 in Th. 13(a). Then, clearly the latter b ound coincidents with its counterpart in Th. 9. F or the pro of of the second part of the corollary , we observe that ρ ( α ) = 0 and that the quan tities c i , d i , and γ i coinciden t with its coun terparts in Th. 9. Moreo ver, remo ving the distribution dep endence by upp er b ounding E (  i ) ≤ 1 reveals that δ i is upp er b ounded b y its coun ter part of Th. 9. Hence, the whole expression on the righ t hand side of Th. 13(b) is upp er b ounded by its coun terpart in Th. 9(b). The following corollary shows the asymptotic b eha vior of the ab o ve theorem. It follows from γ i , δ n , ρ ( α ) → 0 for i, n → ∞ , and α → 0, resp ectively . Corollary 15 L et C b e a c entr oid le arner with maximal false p ositive r ate α satisfying the optimal attack str ate gy. Then for the displac ement of C , denote d by D i , we have: (a) E ( D i ) ≤ ν + α (1 − ν ) (1 − ν )(1 − α ) for al l i (b) V ar( D i ) → 0 for i, n → ∞ , α → 0 . F rom the previous theorem, we can see that for small false p ositive rates α ≈ 0, whic h are common in many applications, e.g., Intrusion Detection (see Sect. 7 for an extensiv e analysis), the b ound approximately equals the one of the previous section, i.e., we hav e E ( D i ) ≤ ν 1 − ν + δ where δ > 0 is a small constant with δ → 0. Inv erting the b ound we obtain the useful formula ν ≥ E ( D i ) 1 + E ( D i ) (17) whic h gives a low er b ound on the minimal ν an adversary has to employ for an attack to succeed. 21 0 1 2 3 4 5 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 i/n D i average−out for various FP levels α α =0.01 α =0.0075 α =0 α =0.005 α =0.002 Figure 7: Theoretical b ehavior of the displacemen t of a centr oid under a p oisoning attac k for diﬀeren t levels of false p ositive protection α . The predicted displacemen t curv e for α = 0 coincides with the one shown in Fig. 5.3. The b ound of Th. 13 is sho wn in Fig. 5.3 for diﬀerent levels of false p ositiv e protection α ∈ [0 , 0 . 025]. W e are especially interested in low p ositiv e rates whic h are common in anomaly detection applications. One can see that muc h of the tigh tness of the b ounds of the previous section is preserved. In the extreme case α = 0 the b ounds coinciden t, as b een sho wn in Cor. 14. 7. Case Study: Application to Intrusion Detection In this section we present the exp erimen tal ev aluation of the developed analytical instru- men ts in the context of a particular computer securit y application: intrusion detection. Cen troid anomaly detection has b een previously used in several in trusion detection systems (e.g., Hofmeyr et al., 1998; Lazarevic et al., 2003; W ang and Stolfo, 2004; Lask ov et al., 2004b; W ang et al., 2005; Rieck and Lasko v, 2006; W ang et al., 2006; Rieck and Lasko v, 2007). After a short presen tation of data collection, prepro cessing and mo del selection, our exp eriments aim at v eriﬁcation of the theoretically obtained gro wth rates for attac k progress as well as computation of constan t factors for sp eciﬁc exploits. 7.1 Data Corpus and Prepro cessing The data to b e used in our case study represents real HTTP traﬃc recorded at F raun- hofer FIRST. W e consider the intermediate granularit y lev el of requests which are the basic application-lay er syn tactic elements of the HTTP proto col. P ack et headers hav e b een stripp ed, and requests spread across m ultiple pac kets hav e b een merged together. The re- sulting b enign dataset consists of 2950 b yte strings containing payloads of inbound HTTP 22 requests. The malicious dataset consists of 69 attack instances from 20 classes generated using the Metasploit penetration testing framew ork 6 . All exploits w ere normalized to matc h the frequent attributes of inno cuous HTTP requests such that the malicious payload pro- vides the only indicator for iden tifying the attac ks. As b yte sequences are not directly suitable for application of mac hine learning algo- rithms, we deploy a k -gram sp ectrum kernel (Leslie et al., 2002; Shaw e-T a ylor and Cris- tianini, 2004) for the computation of the inner pro ducts. T o enable fast comparison of large b yte sequences (a typical sequence length 500-1000 b ytes), eﬃcien t algorithms using sorted arra ys (Riec k and Lasko v, 2008) hav e b een implemented. F urthermore, k ernel v alues are normalized according to k ( x , ¯ x ) 7− → k ( x , ¯ x ) p k ( x , x ) k ( ¯ x , ¯ x ) , (18) to a void a dep endence on the length of a request payload. The resulting inner pro ducts subsequen tly hav e b een pro cessed by an RBF kernel. 7.2 Learning Mo del The feature space selected for our exp eriments dep ends on t wo parameters: the k -gram length and the RBF kernel width σ . Prior to the main exp eriments aimed at the v alidation of prop osed security analysis tec hniques, we inv estigate optimal mo del parameters in our feature space. The parameter range considered is k = 1 , 2 , 3 and σ = 2 − 5 , 2 − 4 , ..., 2 5 . T o carry out mo del selection, we randomly partitioned the inno cuous corpus in to disjoin t training, v alidation and test sets (of sizes 1000, 500 and 500). The training partition is comprised of the inno cuous data only , as the online centroid learner assumes clean training data. The v alidation and test partitions are mixed with 10 attack instances randomly c hosen from diﬀer ent attack classes. 7 F or each partition, diﬀerent online cen troid learner mo dels are trained on a training set and ev aluated on a v alidation and a test sets using the normalized 8 A UC [0 , 0 . 01] as a p erformance measure. F or statistical signiﬁcance, mo del selection is rep eated 1000 times with diﬀeren t randomly drawn partitions. The av erage v alues of the normalized AUC [0 , 0 . 01] for the diﬀerent k v alues on test partitions are giv en in T able 1. It can b e seen that the 3-gram mo del consisten tly shows b etter AUC v alues for b oth the linear and the b est RBF k ernels. W e ha ve chosen the linear kernel for the remaining exp erimen ts, since it allows to carry out computations directly in input space with only a marginal p enalt y in detection accuracy . 7.3 Intrinsic HTTP Data Dimensionalit y Dimensionalit y of training data mak es an imp ortant con tribution to the (in)security of the online centroid learner when using the nearest-out up date rule. Simulations on artiﬁcial data (cf. Section 4.2.3) sho w that the slop e of a linear progress rate of a poisoning attac k increases for larger dimensionalities d . This can b e also explained theoretically (cf. Section B.1) b y 6. http://www.metasploit.com/ 7. The latter requirement reﬂects the goal of anomaly detection to recognize previously unknown attacks. 8. such that an AUC of 1 is the highest achiev able v alue 23 linear b est RBF kernel optimal σ 1-grams 0 . 913 ± 0 . 051 0 . 985 ± 0 . 021 2 − 2 . 5 2-grams 0 . 979 ± 0 . 026 0 . 985 ± 0 . 025 2 − 1 . 5 3-grams 0 . 987 ± 0 . 018 0 . 989 ± 0 . 017 2 − 0 . 5 T able 1: Accuracy of the linear kernel and the b est RBF k ernel as well as the optimal bandwidth σ . the fact that radius of V oronoi cells induced by training data is prop ortional to d p 1 /n , whic h increases with gro wing d . F or the intrusion detection application at hand, the dimensionalit y of the chosen feature space ( k -grams with k = 3) is 256 3 . In view of Th. 16, the dimensionality of the relev ant subspace in which attack takes place is b ounded b y the size of the training data n , which is muc h smaller, in the range of 100 – 1000 for realistic applications. Y et the real progress rate dep ends on the intrinsic dimensionalit y of the data. When the latter is smaller than the size of the training data, an attack er can compute a PCA of the data matrix (Sc h¨ olkopf et al., 1998) and pro ject the original data into a subspace spanned by a smaller n umber of informativ e comp onents. T o determine the intrinsic dimensionality of p ossible training sets drawn from HTTP traﬃc, w e randomly drew 1000 elements from the training set, calculate a linear k ernel matrix in the space of 3-grams and compute its eigenv alue decomp osition. W e then deter- mine the num b er of leading eigen-comp onen ts preserving as a function of the p ercentage of v ariance preserved. The results av eraged o ver 100 repetitions are sho wn in Fig. 8. 0 50 100 150 200 250 300 90 92 94 96 98 100 k variance [%] Figure 8: Intrinsic dimensionality of the embedded HTTP data. The preserv ed v ariance is plotted as a function of the n umber of eigencomp onen ts, k , employ ed for calcu- lation of v ariance (solid blue line). The tube indicates standard deviations. 24 It can b e seen that 250 kernel PCA comp onents are needed to preserve 99% of the v ariance. This implies that, although eﬀectiv e dimensionalit y of HTTP traﬃc is signiﬁcan tly smaller that the num ber of training data p oin ts, it still remains suﬃcien tly high so that the rate of attack progress approac hes 1, which is similar to the simple av erage-out learner. 7.4 Geometrical Constraints of HTTP Data Sev eral technical diﬃculties arising from data geometry hav e to b e ov ercome in launching a p oisoning attack in practice. It turns out, how ev er, that the consideration of the training data geometry provides an attac ker with eﬃcien t to ols for ﬁnding reasonable approximations for the ab ov e men tioned tasks. (1) First, we cannot directly sim ulate a p oisoning attack in the 3-gram input space due to its high dimensionality . An appro ximately equiv alent explicit feature space can b e constructed by applying k ernel PCA to the kernel matrix K . By pruning the eigenv alues “resp onsible” for dimensions with low v ariance one can reduce the size of the feature space to the implicit dimensionality of a problem if the kernel matches the data (Braun et al., 2008). In all subsequent exp eriments we used d = 256 as suggested b y the exp eriments in Section 7.3. (2) Second the crucial normalization condition (18) requires that a solution lies on a unit sphere. 9 Unfortunately , this renders the calculation of an optimal attac k p oint non- con vex. Therefore we pursue the follo wing heuristic pro cedure to enforce normalization: w e explicitly pro ject lo cal solutions (for each V oronoi cell) to a unit sphere, verify their feasibilit y (the radius and the cell constraints), and remov e infeasible p oints from the outer lo op (10). (3) In general one cannot exp ect each feature space vector to corresp ond to a v alid b yte sequence since not all combinations of k -grams can b e “glued” to a v alid b yte sequence. In fact, ﬁnding a sequence with the b est appro ximation to a given k -gram feature vector has b een shown to b e NP-hard (F ogla and Lee, 2006). F ortunately b y the fact that an optimal attac k lies in the span of training data, i.e. Th. 16, we construct an attac k’s b yte sequence by concatenating original sequences of basis p oin ts with rational co eﬃcien ts that appro ximately match the co eﬃcients of the linear com bination. A p otential disadv antage of this metho d is the large increase in the sequence lengths. Large requests are conspicuous and may consume signiﬁcant resources on the attack er’s part. (4) An attac k byte sequence m ust b e em b edded in a v alid HTML proto col frame. Build- ing a v alid HTTP request with arbitrary con tent is, in general, a non-trivial task, esp ecially if it is required that a request do es not cause an error on a serv er. An HTTP request consists of ﬁxed format headers and a v ariable format b o dy . A most straightforw ard wa y to stealthily in tro duce arbitrary conten t is to provide a b o dy in a request whose metho d (e.g., GET) do es not require one. According to an RFC sp eciﬁcation of the HTTP proto col, a request b o dy should b e ignored b y a server in this case. 9. In the absence of normalization, the high v ariabilit y of the byte sequence lengths leads to p oor accuracy of the centroid anomaly detection. 25 7.5 Poisoning A ttack for Finite Horizon Cen troid Learner The analysis carried out in Section 4 shows that an online cen troid learner, in general, do es not pro vide suﬃcient security if an attack er fully con trols the data. Practical eﬃciency of a p oisoning attack, how ever, dep ends on the dimensionalit y and geometry of training data analyzed in the previous section. Theoretical results hav e b een illustrated in simulations on artiﬁcial data presen ted in Section 4.2.3. Exp eriments in this section are intended to verify whether these ﬁndings hold for real attac ks against HTTP applications. Our exp eriments fo cus on the nearest-out learner, as other up date rules can b e easily attack ed with trivial metho ds. W e are now in the p osition to ev aluate the progress rate of a p oisoning attack on real net work traﬃc and exploits. The goal of these exp erimen ts is to verify simulations carried out in Section 4.2.2 on real data. Our exp erimen tal proto col is as follows. W e randomly draw n = 250 training p oints from the inno cuous corpus, calculate the cen ter of mass and ﬁx the radius suc h that the false p ositive rate on the training data is α = 0 . 001. Then we dra w a random instance from eac h of the 20 attack classes, and for eac h of these 20 attac k instances generate a p oisoning attac k as describ ed in Section 7.4. An attack succeeds when the attack p oin t is accepted as inno cuous by a learning algorithm. F or eac h attack instance, the n umber of iterations needed for an attac k to succeed and the resp ective displacement of the cen ter of mass is recorded. Figure 9 sho ws, for eac h attac k instance, the b eha vior of the relativ e displacemen t at the p oin t of success as a function of a n umber of iterations. W e interpolate a “displacement curve” from these p oin twise v alues b y a linear least-squares regression. F or comparison, the theoretical upp er b ounds for the a verage-out and all-in cases are sho wn. Notice that the b ound for the all-in strategy is also almost linear for the small i/n ratios observed in this exp eriment. The observed results conﬁrm that the linear progress rate in the full control scenario can b e attained in practice for real data. Compared to the simulations of Section 7.4, the progress rate of an attac k is approximately half the one for the a verage-out case. Although this somewhat con tradicts our exp ectation that for a high-dimensional space (of the eﬀectiv e dimensionalit y d ∼ 256 as it was found in Section 7.3) the progress rate to the av erage-out case should b e observed, this can b e attributed to m ultiple approximations p erformed in the generation of an attack for real b yte sequences. The practicality of a poisoning attac k is further emphasized by a small num b er of iterations needed for an attack to succeed: from 0 to only 35 percent of the initial n umber of p oin ts in the training data ha v e to b e o v erwritten b y an attac ker. 7.6 Critical T raﬃc Ratios of HTTP Attac ks F or the case of attac k er’s limited con trol, the success of the p oisoning attac k largely dep ends on attac ker’s constrain ts, as shown in the analysis in Sections 5 and 6. The main goal of the exp erimen ts in this section is therefore to in vestigate the impact of p oten tial constrain ts in practice. In particular, we are interested in the impact of the traﬃc ratio ν and the false p ositiv e rate α . The analysis in Section 5 (cf. Theorem 9 and Figure 5.3) sho ws that the displacemen t of a p oisoning attac k is b ounded from ab ov e by a constan t, dep ending on the traﬃc ratio 26 0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.05 0.1 0.15 0.2 0.25 0.3 i/n D i average−out (finite window) average−out (infinite window) nearest−out Figure 9: Empirical displacemen t of the nearest-out cen troid for 20 diﬀeren t exploits (crosses, linear ﬁt sho wn by a red dotted line). Displacement v alues are shown at the point of success for eac h attac k. Theoretical bounds are sho wn for comparison (blue and black lines). ν con trolled by an attac ker. Hence the susceptibilit y of a learner to a particular attack dep ends on the v alue of this constant. If an attack er do es not control a suﬃciently large traﬃc p ortion and the p otential displacement is b ounded b y a constant smaller than the distance from the initial cen ter of mass to the attack p oint, then an attack is bound to fail. T o illustrate this observ ation, w e compute critical traﬃc rates needed for the success of each of the 20 attac k classes in our malicious p o ol. W e randomly draw a 1000-elemental training set from the inno cuous p o ol and calculate its c en ter of mass (in the space of 3-grams). The radius is ﬁxed such the false p ositive rate α = 0 . 001 on inno cuous data is attained. F or each of the 20 attack classes w e compute the class-wise median distance to the cen troid’s b oundary . Using these distance v alues w e calculate the “critical v alue” ν crit b y solving Th. 9(c) for ν (cf. Eq. (17)). The exp erimen ts ha ve b een rep eated 10 times results are shown in T able 2. The results indicate that in order to sub vert a online centroid learner an attack er needs to control from 5 to 20 p ercent of traﬃc. This could b e a signiﬁcant limitation on highly visible sites. Note that an attack er usually aims at earning money by hac king computer systems. How ever generating comp etitive bandwidths at highly visible site is lik ely to driv e the attack er’s cost to exorbitant num b ers. On the other hand, one can see that the traﬃc rate limiting alone cannot b e seen as suﬃcien t protection instrument due to its passive nature. In the following section we in vestigate a diﬀeren t protection sc heme using both traﬃc ratio and the false p ositiv e rate con trol. 27 A ttacks Rel. dist. ν crit AL T-N W ebAdmin Overﬂo w 0 . 058 ± 0 . 002 0 . 055 ± 0 . 002 Apac heChunk edEnco ding 0 . 176 ± 0 . 002 0 . 150 ± 0 . 001 A WStats ConﬁgDir Execution 0 . 067 ± 0 . 002 0 . 063 ± 0 . 002 Badblue Ext Overﬂo w 0 . 168 ± 0 . 002 0 . 144 ± 0 . 001 Barracuda Image Execution 0 . 073 ± 0 . 002 0 . 068 ± 0 . 002 Edirectory Host 0 . 153 ± 0 . 002 0 . 132 ± 0 . 001 IA W ebmail 0 . 178 ± 0 . 002 0 . 151 ± 0 . 001 I IS 5.0 IDQ exploit 0 . 162 ± 0 . 002 0 . 140 ± 0 . 001 P a jax Execute 0 . 107 ± 0 . 002 0 . 097 ± 0 . 002 PEER CAST URL 0 . 163 ± 0 . 002 0 . 140 ± 0 . 001 PHP Include 0 . 097 ± 0 . 002 0 . 088 ± 0 . 002 PHP vBulletin 0 . 176 ± 0 . 002 0 . 150 ± 0 . 001 PHP XML RPC 0 . 172 ± 0 . 002 0 . 147 ± 0 . 001 HTTP tunnel 0 . 160 ± 0 . 002 0 . 138 ± 0 . 001 I IS 4.0 HTR exploit 0 . 176 ± 0 . 002 0 . 149 ± 0 . 002 I IS 5.0 prin ter exploit 0 . 161 ± 0 . 002 0 . 138 ± 0 . 001 I IS unico de attack 0 . 153 ± 0 . 002 0 . 133 ± 0 . 001 I IS w3who exploit 0 . 168 ± 0 . 002 0 . 144 ± 0 . 001 I IS 5.0 W ebD A V exploit 0 . 179 ± 0 . 002 0 . 152 ± 0 . 001 rpro xy exploit 0 . 155 ± 0 . 002 0 . 134 ± 0 . 001 T able 2: Relative distances (in radii) of exploits to the b oundary of a centroid enclosing all training p oin ts and critical v alues of parameter ν . 7.7 Poisoning A ttack against Learner with F alse Positiv e Protection The analysis in Section 5 (cf. Theorem 9 and Figure 5.3) shows that the displacement of a p oisoning attac k is b ounded from ab ov e b y a constant, dep ending on a traﬃc ratio ν and a maximal false p ositiv e rate α . Hence a detection system can b e protected by observing the system’s false p ositive rate and switching oﬀ the online up dates if a deﬁned threshold is exceeded. 7.7.1 Experiment 1: Practicability of F alse Positive Protection Ho wev er in practice the system should b e as silent as p ossible, i.e., an administrator should b e only alarmed if a fatal danger to the system is giv en. W e hence in this section in vestigate ho w sensible the false p ositiv e rate is to small adversarial perturbations of the learner, caused b y p oisoning attack with small ν . Therefore the following exp erimen t inv estigates the rise in the false p ositive rate α as a function of ν . F rom the inno cuous p o ol we randomly drew a 1000-elemental training set on base of which a centroid is calculated. Thereby the radius is ﬁxed to the empirical estimate of the 0 . 001-quan tile of the inno cuous p o ol based on 100 randomly drawn subsamples, i.e., we exp ect the cen troid having a false p ositiv e rate of α = 0 . 001 on the inno cuous p o ol. Moreo ver 28 0 0.05 0.1 0.15 0.2 0.25 0 0.002 0.004 0.006 0.008 0.01 ν α max α max ( ν ) Figure 10: Maximal false p ositiv e rate within 10000 attack iterations as a function of ν (maxim um taken o ver 10 runs). w e randomly drew a second 500-elemen tal training set from the inno cuous p o ol whic h is reserv ed for online training and and a 500-elemental hold out set on base of which a false p ositiv e rate can b e estimated for a giv en cen troid. Then w e iteratively calculated poisoning attac ks with ﬁxed I IS 5.0 W ebD A V exploit as attac k p oin t b y subsequen tly presenting online training points to the cen troid learner whic h are rejected or accepted based on whether they fall within the learner’s radius. F or eac h run of a poisoning attack the false p ositiv rate is observ ed on base of the hold out set. In Fig. 10 we plot for v arious v alues of ν the maximal observed false p ositiv e rate as a function of ν , where the maxim um is taken o ver all attack iterations and 10 runs. One can see from the plot that α = 0 . 005 is a reasonable threshold in our setting to ensure the systems’s silentness. 0 2 4 6 8 10 0 0.05 0.1 0.15 0.2 i/n D i D crit ν =0.16 ν =0.14 ν =0.1 ν =0.05 Figure 11: A simulation of a p oisoning attack under limited con trol. 29 7.7.2 Experiment 2: A tt ack Simula tion for F alse Positive Protection In the previous exp eriment w e ha ve seen that α = 0 . 005 is a reasonable threshold for a false p ositive protection to ensure a systems silentness. W e in this section illustrate that the critical v alues from Section 7.6 computed on base of Th. 9 for maximal false p ositiv e rate of α = 0 . 005 still giv e a goo d approximation of the true impact of a p oisoning attack. W e ﬁx a particular exploit in our malicious corpus (I IS W ebD A V 5.0 exploit) and run a p oisoning attac k against the av erage-out centroid for v arious v alues of ν ∈ [0 . 05 , 0 . 10 , 0 . 14 , 0 . 16], recording the actual displacemen t curves. One can see from Fig. 11 that the attack suc- ceeds for ν = 0 . 16 but fails to reac h the required relativ e displacement of D crit = 0 . 18 for ν = 0 . 14. The theoretically computed critical traﬃc ratio for this attack according to T able 2 is ν crit = 0 . 152. The exp eriment shows that the derived b ounds are surprisingly tigh t in practice. 7.7.3 Implement a tion of Poisoning Protection In Section 5 we hav e seen, that an attack er’s impact on corrupting the training data highly dep ends on the fraction of adv ersarial p oints in the training data stream. This implies that a high amount of inno cuous training p oints constantly has to come in. In Section 6 we ha ve seen, that w e can secure the learner b y setting a threshold on the false p ositive rate α . Exceeding the latter enforces further defense pro cesses such as switching oﬀ the online training process. Hence an conﬁden t estimation of α has to b e at hand. Ho w can w e ac hieve the latter? In practice, this can e.g. b e done b y caching the training data. When the cache exceeds a certain v alue at which we ha ve a conﬁden t estimation of α (e.g., after 24 hours), the cac hed training data can b e applied to the learner. Since in applications including intrusion detection, we usually deal with a very high amount of training data, a conﬁden t estimation is already p ossible after short time p erio d. 8. Discussion and Conclusions Understanding of security prop erties of learning algorithms is essen tial for their protection against abuse. The latter can take place when learning is used in applications with c om- p etitiv e interests at stake, e.g., security monitoring, games, spam protection, reputation systems, etc. Certain securit y prop erties of a learning algorithm must b e pr ove d in order to claim its immunit y to abuse. T o this end, we hav e developed a metho dology for securit y analysis and applied it for a sp eciﬁc scenario of online centroid anomaly detection. The results of our analysis highligh t conditions under whic h an attac ker’s eﬀort to sub vert this algorithm is prohibitively high. Sev eral issues discussed in this contribution hav e app eared in related work alb eit not in the area of anomaly detection. Perhaps the most consummate treatment of learning under an adversarial impact has b een carried out b y Dalvi et al. (2004). In this work, Ba yesian classiﬁcation is analyzed for robustness against adv ersarial impact. The choice of their classiﬁer is motiv ated b y widespread application of the naiv e Ba y es classiﬁcation in the domain of spam detection where real examples of adv ersarial impact hav e b een observ ed for a long time. The adversarial classiﬁcation is considered as a game b etw een an attack er and 30 a learner. Due to the complexit y of analysis, only one mo ve by each party can b e analyzed. Similar to our approac h, Dalvi et al. (2004) formalize the problem b y deﬁning cost functions of an attac ker and a learner (Step 1) and determine an optimal adv ersarial strategy (Step 3). Although the attac ker’s constrain ts are not explicitly treated theoretically , several scenarios using speciﬁc constrain ts ha ve b een tested exp erimen tally . No analysis of the attac ker’s gain is carried out; instead, the learner’s direct resp onse to adversarial impact is considered. A somewhat related approac h has been developed for handling w orst-case random noise, e.g., random feature deletion (Glob erson and Row eis, 2006; Dek el and Shamir, 2008). Sim- ilar to Dalvi et al. (2004), b oth of these metho ds construct a classiﬁer that automatically reacts to the worst-case noise or, equiv alently , the optimal adv ersarial strategy . In b oth metho ds, the learning problem is form ulated as a large-margin classiﬁcation using a sp e- cially constructed risk function. An imp ortan t role in this approach is pla yed b y the con- sideration of constraints (Step 2), e.g., in the form of the maximal n umber of corruptible features. Although these approac hes do not quan titatively analyze attac ker’s gain, (Dek el and Shamir, 2008) contains an interesting learning-theoretic argument that relates classiﬁ- cation accuracy , sparseness, and robustness against adv ersarial noise. T o summarize, we believe that despite recent evidence of p ossible attac ks against ma- c hine learning and the currently lacking theoretical foundations for learning under adv er- sarial impact, machine learning algorithms c an b e protected against such impact. The key to such protection lies in quan titative analysis of securit y of mac hine learning. W e hav e sho wn that such analysis can b e rigorously carried out for sp eciﬁc algorithms and attacks. F urther work should extend such analysis to more complex learning algorithms and a wider attac k sp ectrum. Ac kno wledgments The authors wish to thank Ulf Brefeld, Konrad Rieck, V o jtech F ranc, Peter Bartlett and Klaus-Rob ert M ¨ uller for fruitful discussions and helpful commen ts. F urthermore w e thank Konrad Rieck for providing the net work traﬃc. This work was supp orted in part by the German Bundesministerium f ¨ ur Bildung und F orsch ung (BMBF) under the pro ject RE- MIND (FKZ 01-IS07007A), by the German Academic Exchange Service, and by the FP7- ICT Programme of the Europ ean Communit y , under the P ASCAL2 Net work of Excellence, ICT-216886. App endix A. Notation Summary In this pap er we use the follo wing notational con ven tions. C , r , c cen troid C with radius r and center c i i -th attack iteration, i ∈ N 0 x i , X i cen ter of cen troid in i -th attac k iteration A attack p oint a attack direction v ector 31 D i i -th relative displacement of a cen troid in radii into direction of a n n umber of training patterns of centroid f function of H → H giving an attack strat- egy ν fraction of adversarial training p oints B i Bernoulli v ariable  i ,  i i.i.d. noise α false alarm rate I S indicator function of a set S App endix B. Auxiliary Material and Pro ofs B.1 Auxiliary Material for Section 4 B.1.1 Representer Theorem for Optimal Greedy A tt ack First, w e show why the attac k eﬃciency cannot b e increased b eyond dimensions with d ≥ n + 1. This follows from the fact that the optimal attack lies in the span of the working set p oin ts and the attack v ector. The follo wing representer theorem allows for “kernelization” of the optimal greedy attac k. Theorem 16 Ther e exists an optimal solution of pr oblem (11) satisfying x ∗ i ∈ span( a , x 1 , ..., x n ) . (19) Pro of The Lagrangian of optimization problem (11) is giv en by: L ( x , α , β ) = − ( x − x i ) · a + n X j =1 α j (2( x j − x i ) · x − x j · x j + x i · x i ) + β   x · x − 2 n n X j =1 x · x j + 1 n 2 n X j,k =1 x j · x k − r 2   Since the feasible set of problem (11) is bounded b y the spherical constraint and is not empt y ( x i trivially is con tained in the feasible set), there exists at least one optimal solution x ∗ i to the primal. F or optimal x ∗ i , α ∗ and β ∗ , we ha ve the follo wing ﬁrst order optimality conditions δ L δ x = 0 : − a − 1 n n X j =1 x j + 2 n X j =1 α ∗ j ( x j − x i ) + β ∗   2 x ∗ i − 2 n n X j =1 x j   = 0 . (20) If β ∗ 6 = 0 the latter equation can b e resolv ed for x ∗ i leading to: x ∗ i = 1 2 β ∗ a + n X j =1  1 2 β ∗ n − α ∗ j β ∗ + 1 n  x j + 1 β ∗ n X j =1 α ∗ j x i . 32 F rom the latter equation we see that x is contained in S := span( x 1 , ..., x n and a ). No w assume β ∗ = 0 and x ∗ i / ∈ S . Basically the idea of the follo wing reasoning is to use x ∗ i to construct an optimal p oint whic h is contained in S . A t ﬁrst, since β ∗ = 0, we see from Eq. (20) that a is contained in the subspace S := span( x 1 , ..., x n ). Hence the ob jective, ( x − x i ) · a , only dep ends on the optimal x via inner pro ducts with the data x i . The same naturally holds for the constraints. Hence b oth, the ob jectiv e v alue and the constraints, are in v arian t under the pro jection of x ∗ i on to S , denoted by P . Hence P ( x ∗ i ) also is an optimal p oin t. Moreov er by construction P ( x ∗ i ) ∈ S = span( x ∗ 1 , ..., x ∗ n ). B.1.2 Theoretical Anal ysis for the Optimal Greedy A tt ack The dep endence of an attac k’s eﬀectiveness on the data dimensionality results from the geometry of V oronoi cells. In tuitively , the displacement at a single iteration dep ends on the size of the largest V oronoi cell in a current working set. Although it is hard to derive a precise estimate on the latter, the following “a verage-case” argument sheds some ligh t on the attack’s b eha vior, esp ecially since it is the av erage-case geometry of the working set that determines the ov erall – as opposed to a single iteration – attac k progress. Consider a simpliﬁed case where each of the V oronoi cells C j constitutes a ball of radius r centered at a data p oint x j , j = 1 , . . . , n . Clearly , the greedy attack will results in a progress of r /n (we will mo ve one of the p oints by r but the center’s displacement will b e discoun ted b y 1 /n ). W e will now use the relationships b etw een the volumes of balls in R d to relate r , R and d . The volume of each V oronoi cell C j is given by V ol( C j ) = π d 2 r d Γ  d 2 + 1  . Lik ewise, the v olume of the h yp ersphere S of radius R is V ol( S ) = π d 2 R d Γ  d 2 + 1  . Assuming that the V oronoi cells are “tigh tly pack ed” in S , we obtain V ol( S ) ≈ n V ol( C j ) . Hence we conclude that r ≈ d r 1 n R. One can see that the attac ker’s gain, approximately represented by the cell radius r , is a constan t fraction of the threshold R , which explains the linear progress of the p oisoning attac k. The slop e of this linear dependence is con trolled b y tw o opp osing factors: the size of the training data decreases the attack sp eed whereas the intrinsic dimensionalit y of the feature space increases it. Both factors dep end on ﬁxed parameters of the learning problem 33 and cannot b e controlled by an algorithm. In the limit, when d approaches n (the eﬀective dimension is limited by the training data set according to Th. 16) the attac k progress rate is approximately describ ed by the function n q 1 n whic h approaches 1 with increasing n . B.2 Pro ofs of Section 5 Prop osition 17 (Geometric series) L et ( s ) i ∈ N 0 b e a se quenc e of r e al numb ers satisfying s 0 = 0 and s i +1 = q s i + p (or s i +1 ≤ q s i + p or s i +1 ≥ q s i + p ) for some p, q > 0 . Then it holds: s i = p 1 − q i 1 − q , (and s i ≤ p 1 − q i 1 − q or s i ≥ p 1 − q i 1 − q ) , (21) r esp e ctively. Pro of (a) W e prov e part (a) of the theorem b y induction ov er i ∈ N 0 , the case of i = 0 b eing ob vious. In the inductive step we show that if Eq. (21) holds for an arbitrary ﬁxed i it also holds for i + 1: s i +1 = q s i + p = q  p 1 − q i 1 − q  + p = p  q 1 − q i 1 − q + 1  = p  q − q i +1 + 1 − q 1 − q  = p  1 − q i +1 1 − q  . (b) The pro of of part (b) is analogous. Pro of of Th. 9(b) Multiplying b oth sides of Eq. (15) with a and substituting D i = X i · a results in D i +1 =  1 − 1 − B i n  D i + B i n + (1 − B i ) n  i · a . Inserting B 2 i = B i and B i (1 − B i ) = 0, which holds b ecause B i is Bernoulli, into the latter equation, we hav e: D 2 i +1 =  1 − 2 1 − B i n + 1 − B i n 2  D 2 i + B i n 2 + (1 − B i ) n 2 k  i · a k 2 +2 B i n D i +2(1 − B i )(1 − 1 n ) D i  i · a . T aking the exp ectation on the latter equation, and noting that by Axiom 6  i and D i are indep enden t, we hav e: E  D 2 i +1  =  1 − 1 − ν n  2 − 1 n  E  D 2 i  + 2 ν n E ( D i ) + ν n 2 + 1 − ν n 2 E ( k  i · a k 2 ) (1) ≤  1 − 1 − ν n  2 − 1 n  E  D 2 i  + 2 ν n E ( D i ) + 1 n 2 (22) where (1) holds b ecause b y Axiom 6 we hav e k  i k 2 ≤ r and by Def. 7 k a k = R , R = 1. Inserting the result of (a) in the latter equation results in the following recursive formula: E  D 2 i +1  ≤  1 − 1 − ν n  2 − 1 n  E  D 2 i  + 2(1 − c i ) ν n ν 1 − ν + 1 n 2 . 34 By the formula of the geometric series, i.e., by Prop.17, w e hav e: E  D 2 i  ≤  2(1 − c i ) ν n ν 1 − ν + 1 n 2  1 − d i 1 − ν n  2 − 1 n  , denoting d i :=  1 − 1 − ν n  2 − 1 n  i . F urthermore b y some algebra E  D 2 i  ≤ (1 − c i )(1 − d i ) 1 − 1 2 n ν 2 (1 − ν ) 2 + 1 − d i (2 n − 1)(1 − ν ) . (23) W e will need the auxiliary formula (1 − c i )(1 − d i ) 1 − 1 2 n − (1 − c i ) 2 ≤ 1 2 n − 1 + c i − d i , (24) whic h can be v eriﬁed by some more algebra and employing d i < c i . W e ﬁnally conclude V ar( D i ) = E ( D 2 i ) − ( E ( D i )) 2 Th.13( a ); Eq.(23) ≤ (1 − c i )(1 − d i ) 1 − 1 2 n − (1 − c i ) 2 !  ν 1 − ν  2 + 1 − d i (2 n − 1)(1 − ν ) 2 Eq.(24) ≤ γ i  ν 1 − ν  2 + δ n where γ i := c i − d i and δ n := ν 2 +(1 − d i ) (2 n − 1)(1 − ν ) 2 . This completes the proof. B.3 Pro ofs of Section 6 Lemma 18 L et C b e a pr ote cte d online c entr oid le arner satisfying the optimal attack str at- e gy. Then we have: (a) 0 ≤ E  I {k  i − X i k >r } D q i  ≤ αE ( D q i ) , q = 1 , 2 (b) 0 ≤ E  I {k  i − X i k≤ r }  i  ≤ α (c) E  I {k  i − X i k≤ r }  i D i  ≤ αE ( D i ) . Pro of (a) Let q = 1 or q = 2. Since  i is indep enden t of X i (and hence of D i ), we hav e E  i  I {k  i − X i k >r } D q i  = ( D i ) q E  i  I {k  i − X i k >r }  . Hence by Ax. 11 E  i  I {k  i − X i k >r } D q i  = 0 if e ( X i ) := E  i  I {k  i − X i k >r }  > α , and 0 ≤ E  i  I {k  i − X i k >r } D q i  ≤ α if e ( X i ) ≤ α . 35 By the symmetry of  i w e conclude statemen t (a). T aking the full exp ectation E = E X i E  i on the latter expression yields the statement. (b) W e denote I ≤ := I {k  i − X i k≤ r } and I > := I {k  i − X i k >r } . Since it holds E ( I ≤  i ) + E ( I >  i ) = E (( I ≤ + I > )  i ) = E (  i ) = 0 , w e conclude E ( I ≤  i ) = − E ( I >  i ) = E ( I > ( −  i )) (1) ≤ α , where (1) holds b ecause ||  i || ≤ 1 and by Ax. 11 w e hav e E ( I > ) ≤ α . F urthermore E ( I ≤  i ) ≥ 0 is clear. (c) The pro of of (c) is analogous to that of (a) and (b). Pro of of Th. 13 (a) By Ax. 11 w e hav e D i +1 = max  0 , D i + 1 n  B i ( f ( X i ) − X i ) + (1 − B i ) I {k  i − X i k≤ r } (  i − X i )  · a  . (25) By Prop. 12 an optimal attac k strategy can b e deﬁned by f ( x ) = x + a . Inserting the latter equation into Eq. (25), using D i Def . = X i · a , and taking the exp ectation, w e hav e E ( D i +1 ) = E  max  0 , D i + 1 n  B i + (1 − B i ) I {k  i − X i k≤ r } (  i − D i )   , (26) denoting  i =  i · a . By the symmetry of  i the exp ectation can b e mov ed inside the maxim um, hence the latter equation can b e rewritten as E ( D i +1 ) ≤  1 − 1 − ν n  E ( D i ) + ν n (27) + 1 − ν n  E  I {k  i − X i k >r } D i  + E  I {k  i − X i k≤ r }  i  . Inserting the inequalities (a) and (b) of Lemma 18 in to the ab ov e equation results in: E ( D i +1 ) ≤  1 − 1 − ν n  E ( D i ) + ν n + 1 − ν n ( αE ( D i ) + α ) =  1 − (1 − ν )(1 − α ) n  E ( D i ) + ν + α (1 − ν ) n . By the formula of the geometric series, i.e., Prop. 17, we ha ve E ( D i +1 ) ≤ (1 − c i ) ν + α (1 − ν ) (1 − ν )(1 − α ) (28) 36 where c i =  1 − (1 − ν )(1 − α ) n  i . Moreov er w e hav e E ( D i +1 ) ≥ (1 − b i ) ν 1 − ν , (29) where b i =  1 − 1 − ν n  i , by analogous reasoning. In a sk etch we sho w that by starting at Eq. (26), and subsequen tly applying Jensen’s inequalit y , the lo wer b ounds of Lemma 18 and the formula of the geometric series. Since b i ≤ c i w e conclude E ( D i +1 ) ≥ (1 − c i ) ν 1 − ν . (30) (b) Rearranging terms in Eq. (25), we hav e D i +1 ≤ max  0 ,  1 − 1 − B i n  D i + B i n + 1 − B i n I {k  i − X i k≤ r }  i + 1 − B i n I {k  i − X i k >r } D i  Squaring the latter equation at b oth sides and using that D i , I {k  i − X i k≤ r } , and I {k  i − X i k >r } are binary-v alued, yields D 2 i +1 ≤  1 − 1 − B i n  2 − 1 n  )  D 2 i + 2 B i n D i +  1 − B i n  2 − 1 n  I {k  i − X i k >r } D i +2 1 − B i n  1 − 1 n  I {k  i − X i k≤ r }  i D i + 1 − B i n 2 I {k  i − X i k≤ r }  2 i + B i n 2 . T aking expectation on the ab o ve equation, b y Lemma 18, w e hav e E ( D 2 i +1 ) ≤  1 − 1 − ν n  2 − 1 n )(1 − α )  E ( D 2 i ) +2  ν n + α 1 − ν n  1 − 1 n  E ( D i ) + ν + (1 − ν ) E (  2 i ) n 2 . W e are now in an equiv alent situation as in the pro of of Th. 8, righ t after Eq. (22). Similary , w e insert the result of (a) into the ab ov e equation, obtaining E ( D 2 i +1 ) ≤  1 − 1 − ν n  2 − 1 n )(1 − α )  E ( D 2 i ) +2  ν n + α 1 − ν n  1 − 1 n  (1 − c i ) ν + α (1 − ν ) (1 − ν )(1 − α ) + ν + (1 − ν ) E (  2 i ) n 2 ≤  1 − 1 − ν n  2 − 1 n  (1 − α )  E ( D 2 i ) + 2(1 − c i ) ( ν + α (1 − ν )) 2 n (1 − ν )(1 − α ) + ν + (1 − ν ) E (  2 i ) n 2 37 By the formula of the geometric series we obtain E ( D 2 i ) ≤  2(1 − c i ) ( ν + α (1 − ν )) 2 n (1 − ν )(1 − α ) + ν + (1 − ν ) E (  2 i ) n 2  1 − d i 1 − ν n (2 − 1 n )(1 − α ) ≤ (1 − c i )(1 − d i )( ν + α (1 − ν )) 2 (1 − 1 2 n )(1 − ν ) 2 (1 − α ) 2 + (1 − d i )( ν + (1 − ν ) E (  2 i )) (2 n − 1)(1 − ν )(1 − α ) , (31) where d i =  1 − 1 − ν n (2 − 1 n )(1 − α )  i . W e ﬁnally conclude V ar( D i ) = E ( D 2 i ) − ( E ( D i )) 2 (30) , (31) ≤ (1 − c i )(1 − d i )( ν + α (1 − ν )) 2 (1 − 1 2 n )(1 − ν ) 2 (1 − α ) 2 + (1 − d i )( ν + (1 − ν ) E (  2 i )) (2 n − 1)(1 − ν )(1 − α ) − (1 − c i ) 2 ν 2 (1 − ν ) 2 (1) ≤ γ i ν 2 (1 − α ) 2 (1 − ν ) 2 + ρ ( α ) + δ n deﬁning γ i = c i − d i , ρ ( α ) = α (1 − c i )(1 − d i )(2 ν (1 − α )+ α ) (1 − 1 2 n )(1 − ν ) 2 (1 − α ) 2 , and δ n = (1 − d i )( ν +(1 − ν ) E (  2 i )) (2 n − 1)(1 − ν )(1 − α ) , where (1) can b e veriﬁed employing some algebra and using the auxiliary formula Eq. (24), which holds for all 0 < d i < c i < 1. This completes the proof of (b). Statemen ts (c) and (d) are easily derived from (a) and (b) b y noting hat 0 ≤ c i < 1, c i → 1 for i → ∞ and δ ( n ) → 0 for n → ∞ . This completes the pro of of the theorem. References D. Angluin and P . Laird. Learning from noisy examples. Machine L e arning , 2(4):434–470, 1988. P . Auer. Learning nested diﬀerences in the presence of malicious noise. The or etic al Computer Scienc e , 185(1):159–175, 1997. M. Bailey , J. Ob erheide, J. Andersen, Z. M. Mao, F. Jahanian, and J. Nazario. Automated classiﬁcation and analysis of internet malware. In R e c ent A danc es in Intrusion Dete ction (RAID) , pages 178–197, 2007. M. Barreno, B. Nelson, R. Sears, A. Joseph, and J. Tygar. Can machine learning b e secure? In ACM Symp osium on Information, Computer and Communic ation Se curity , pages 16– 25, 2006. M. Barreno, P . L. Bartlett, F. J. Chi, A. D. Joseph, B. Nelson, B. I. Rubinstein, U. Saini, and J. D. Tygar. Op en problems in the security of learning. In AISe c ’08: Pr o c e e dings of the 1st ACM workshop on Workshop on AISe c , pages 19–26, New Y ork, NY, USA, 2008. A CM. ISBN 978-1-60558-291-7. doi: h ttp://doi.acm.org/10.1145/1456377.1456382. M. L. Braun, J. Buhmann, and K.-R. M ¨ uller. On relev ant dimensions in k ernel feature spaces. Journal of Machine L e arning R ese ar ch , 9:1875–1908, Aug 2008. 38 N. H. Bsc hout y , N. Eiron, and E. Kushilevitz. P AC learning with nast y noise. In A lgorithmic L e arning The ory (AL T 1999) , pages 206–218, 1999. N. Dalvi, P . Domingos, M. Sumit, and S. D. V erma. Adversarial classiﬁcation. In In KDD , pages 99–108. ACM Press, 2004. O. Dekel and O. Shamir. Learning to classify with missing and corrupted features. In International Confer enc e on Machine L e arning (ICML) , pages 216–223, 2008. P . F ogla and W. Lee. Ev ading netw ork anomaly detection systems: formal reasoning and practical techniques. In ACM Confer enc e on Computer and Communic ations Se curity , pages 59–68, 2006. P . F ogla, M. Sharif, R. P erdisci, O. Kolesniko v, and W. Lee. Polymorphic blending attacks. In Pr o c. of USENIX Se curity Symp osium , pages 241–256, 2006. S. F orrest, S. Hofmeyr, A. Somay a ji, and T. Longstaﬀ. A sense of self for unix pro cesses. In Pr o c. of IEEE Symp osium on Se curity and Privacy , pages 120–128, Oakland, CA, USA, 1996. URL cs.unm.edu/ ~ {}forrest/publications/ieee- sp- 96- unix.pdf . A. Glob erson and S. Ro weis. Nigh tmare at test time: Robust learning by feature deletion. In International Confer enc e on Machine L e arning (ICML) , pages 353–360, 2006. S. Hofmeyr, S. F orrest, and A. Somay a ji. Intrusion detection using sequences of system calls. Journal of Computer Se curity , 6(3):151–180, 1998. M. Kearns and M. Li. Learning in the presence of malicious errors. SIAM Journal on Computing , 22(4):807–837, 1993. P . Lasko v and M. Kloft. A framew ork for quan titativ e security analysis of mac hine learning. In D. Balfanz and J. Staddon, editors, AISe c , pages 1–4. ACM, 2009. ISBN 978-1-60558- 781-3. P . Lasko v, C. Sch¨ afer, and I. Kotenko. Intrusion detection in unlab eled data with quarter- sphere supp ort vector mac hines. In Dete ction of Intrusions and Malwar e, and V ulner a- bility Assessment, Pr o c. of DIMV A Confer enc e , pages 71–82, 2004a. P . Lasko v, C. Sch¨ afer, I. Kotenko, and K.-R. M ¨ uller. Intrusion detection in unlab eled data with quarter-sphere support vector machines (extended version). Pr axis der Informa- tionsver arb eitung und Kommunikation , 27:228–236, 2004b. P . Lasko v, C. Gehl, S. Kr ¨ uger, and K. R. M ¨ uller. Incremen tal supp ort vector learning: Analysis, implementation and applications. Journal of Machine L e arning R ese ar ch , 7: 1909–1936, Sept. 2006. A. Lazarevic, L. Ertoz, V. Kumar, A. Ozgur, and J. Sriv asta v a. A comparativ e study of anomaly detection schemes in netw ork intrusion detection. In Pr o c. of SIAM International Confer enc e on Data Mining (SDM) , 2003. C. Leslie, E. Eskin, and W. Noble. The spectrum k ernel: A string kernel for SVM protein classiﬁcation. In Pr o c. Paciﬁc Symp. Bio c omputing , pages 564–575, 2002. 39 Z. Li, M. Sandhi, Y. Chen, M.-Y. Kao, and B. Cha vez. Hamsa: fast signature generation for zero-da y p olymorphic worms with prov able attac k resilience. In ie e esp , pages 32–47, 2006. N. Littlestone. Learning quickly when irrelev an t attributes ab ound: A new linear threshold algorithm. Machine L e arning , 2:285–318, 1988. D. Lowd and C. Meek. Go o d word attacks on statistical spam ﬁlters. In Pr o c e e dings of the 11 th A CM SIGKDD International Confer enc e on Know le dge Disc overy and Data Mining , pages 641–647, 2005a. D. Lo wd and C. Meek. Adversarial learning. In Confer enc e on Email and Anti-Sp am , 2005b. M. Mark ou and S. Singh. Nov elt y detection: a review – part 1: statistical approaches. Signal Pr o c essing , 83:2481–2497, 2003a. M. Markou and S. Singh. No velt y detection: a review – part 2: neural net work based approac hes. Signal Pr o c essing , 83:2499–2521, 2003b. L. Martein and S. Sc haible. On solving a linear program with one quadratic constraint. De cisions in Ec onomics and Financ e , 10:75–90, 2005. K.-R. M ¨ uller, S. Mik a, G. R¨ atsch, K. Tsuda, and B. Sch¨ olkopf. An in tro duction to k ernel- based learning algorithms. IEEE Neur al Networks , 12(2):181–201, May 2001. A. Nairac, T. N., R. Carr, S. King, P . Cowley , and L. T arassenko. A system for the analysis fo jet vibration data. Inte gr ate d Computer-Aide d Engine ering , 1999. B. Nelson and A. D. Joseph. Bounding an attac k’s complexit y for a simple learning mo del. In Pr o c. of the First Workshop on T ackling Computer Systems Pr oblems with Machine L e arning T e chniques (SysML) , Sain t-Malo, F rance, 2006. B. Nelson, M. Barreno, F. Chi, A. Joseph, B. Rubinstein, U. Saini, C. Sutton, J. Tygar, and K. Xia. Exploiting machine learning to subv ert y our spam ﬁlter. In Pr o c e e dings of the First USENIX Workshop on L ar ge-Sc ale Exploits and Emer gent Thr e ats (LEET’08) , 2008. J. Newsome, B. Karp, and D. Song. P aragraph: Th warting signature learning b y training maliciously . In R e c ent A danc es in Intrusion Dete ction (RAID) , pages 81–105, 2006. E. P arzen. On estimation of probabilit y densit y function and mode. Annals of Mathematic al Statistics , 33:1065–1076, 1962. R. Perdisci, D. Dagon, W. Lee, P . F ogla, and M. Sharif. Misleading worm signature gen- erators using delib erate noise injection. In Pr o c. of IEEE Symp osium on Se curity and Privacy , pages 17–31, 2006. W. Polonik. Measuring mass concen tration and estimating densit y contour clusters – an excess mass approach. Annals of Statistics , 23:855–881, 1995. 40 S. Ra jasegarar, C. Leckie, M. Palanisw ami, and J. Bezdek. Quarter sphere based distrib uted anomaly detection in wireless sensor netw orks. In IEEE International Confer enc e on Communic ations (ICC) , pages 3864–3869, 2007. K. Rieck and P . Lask ov. Detecting unknown net work attac ks using language mo dels. In Dete ction of Intrusions and Malwar e, and V ulner ability Assessment, Pr o c. of 3r d DIMV A Confer enc e , LNCS, pages 74–90, July 2006. K. Rieck and P . Lasko v. Language mo dels for detection of unknown attac ks in netw ork traﬃc. Journal in Computer Vir olo gy , 2(4):243–256, 2007. K. Riec k and P . Lask ov. Linear-time computation of similarit y measures for sequen tial data. Journal of Machine L e arning R ese ar ch , 9(Jan):23–48, 2008. K. Rieck, T. Holz, C. Willems, P . D ¨ ussel, and P . Lask o v. Learning and classiﬁcation of malw are b eha vior. In Dete ction of Intrusions and Malwar e, and V ulner ability Assessment, Pr o c. of 5th DIMV A Confer enc e , LNCS, pages 108–125, 2008. B. Sch¨ olk opf and A. Smola. L e arning with Kernels . MIT Press, Cam bridge, MA, 2002. B. Sc h¨ olkopf, A. Smola, and K.-R. M ¨ uller. Nonlinear comp onent analysis as a kernel eigen- v alue problem. Neur al Computation , 10:1299–1319, 1998. B. Sch¨ olkopf, J. Platt, J. Sha we-T a ylor, A. Smola, and R. Williamson. Estimating the supp ort of a high-dimensional distribution. Neur al Computation , 13(7):1443–1471, 2001. J. Sha we-T a ylor and N. Cristianini. Kernel metho ds for p attern analysis . Cambridge Uni- v ersity Press, 2004. I. Steinw art, D. Hush, and C. Scov el. A classiﬁcation framew ork for anomaly detection. Journal of Machine L e arning R ese ar ch , 6:211–232, 2005. M. Sugiy ama, M. Krauledat, and K.-R. M ¨ uller. Cov ariate shift adaptation b y imp ortance w eighted cross v alidation. Journal of Machine L e arning R ese ar ch , 8:1027–1061, 2007. D. T ax and R. Duin. Data domain description b y supp ort v ectors. In M. V erleysen, editor, Pr o c. ESANN , pages 251–256, Brussels, 1999a. D. F acto Press. D. T ax and R. Duin. Supp ort vector domain description. Pattern R e c o gnition L etters , 20 (11–13):1191–1199, 1999b. A. Tsybak ov. On nonparametric estimation of density level sets. Annals of Statistics , 25: 948–969, 1997. C. v an de Panne. Programming with a quadratic constraint. Management Scienc e , 12: 798–815, 1966. V. V apnik. Statistic al L e arning The ory . Wiley , New Y ork, 1998. K. W ang and S. Stolfo. Anomalous payload-based net work intrusion detection. In R e c ent A danc es in Intrusion Dete ction (RAID) , pages 203–222, 2004. 41 K. W ang, G. Cretu, and S. Stolfo. Anomalous payload-based worm detection and signature generation. In R e c ent A danc es in Intrusion Dete ction (RAID) , 2005. K. W ang, J. P arekh, and S. Stolfo. Anagram: A con tent anomaly detector resistant to mimicry attack. In R e c ent A danc es in Intrusion Dete ction (RAID) , pages 226–248, 2006. C. W arrender, S. F orrest, and B. P earlmutter. Detecting intrusions using system calls: alternativ e data methods. In Pr o c. of IEEE Symp osium on Se curity and Privacy , pages 133–145, 1999. D.-Y. Y eung and C. Cho w. P arzen-window netw ork in trusion detectors. In Sixte enth Inter- national Confer enc e on Pattern R e c o gnition (ICPR) , pages 385–388, 2002. 42

Security Analysis of Online Centroid Anomaly Detection

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment