Data Driven Exploratory Attacks on Black Box Classifiers in Adversarial Domains

Data Dri ven Exploratory Attacks on Black Box Classiﬁers in Adv ersarial Domains T egjyot Singh Sethi a, ∗ , Mehmed Kantardzic a a Data Mining Lab, University of Louisville, Louisville, USA Abstract While modern day web applications aim to create impact at the civilization lev el, they have become vulnerable to adversarial acti vity , where the next cyber -attack can take any shape and can originate from anywhere. The increasing scale and sophistication of attacks, has prompted the need for a data driv en solution, with machine learning forming the core of many cybersecurity systems. Machine learning was not designed with security in mind, and the essential assumption of stationarity , requiring that the training and testing data follow similar distributions, is violated in an adversarial domain. In this paper, an adversary’ s vie w point of a classiﬁcation based system, is presented. Based on a formal adversarial model, the Seed-Explor e-Exploit framework is presented, for simulating the generation of data dri ven and reverse engineering attacks on classiﬁers. Experimental ev aluation, on 10 real world datasets and using the Google Cloud Prediction Platform, demonstrates the innate vulnerability of classiﬁers and the ease with which e vasion can be carried out, without any explicit information about the classiﬁer type, the training data or the application domain. The proposed frame work, algorithms and empirical ev aluation, serve as a white hat analysis of the vulnerabilities, and aim to foster the dev elopment of secure machine learning frame works. K e ywor ds: Adversarial machine learning, re verse engineering, black box attacks, classiﬁcation, data di versity, cybersecurity . 1. Introduction The growing scale and reach of modern day web ap- plications has increased its reliance on machine learning techniques, for providing security . Con ventional secu- rity mechanisms of ﬁrewalls and rule-based black and white lists, cannot e ﬀ ectiv ely thwart ev olving attacks at a large scale (Saha and Sanyal, 2014). As such, the use of data driv en machine learning techniques in cy- bersecurity applications, has found widespread accep- tance and success (Group et al., 2013). Whether it be for outlier detection for network intrusion analysis (Za- mani and Mov ahedi, 2013), biometric authentication us- ing supervised classiﬁcation (DSouza, 2014), or for un- supervised clustering of fraudulent clicks (W algampaya and Kantardzic, 2011), the use of machine learning in cybersecurity domains is ubiquitous. Howe ver , during this era of increased reliance on machine learning mod- els, the vulnerabilities of the learning process itself have ∗ Corresponding author . Email addr esses: tegjyotsingh.sethi@louisville.edu (T egjyot Singh Sethi ), mehmedkantardzic@louisville.edu (Mehmed Kantardzic) mostly been overlook ed. Machine learning operates un- der the assumption of stationarity , i.e. the training and the testing distributions are assumed to be identically and independently distributed (IID) ( ˇ Zliobait ˙ e, 2010). This assumption is often violated in an adversarial set- ting, as adversaries gain nothing by generating samples which are block ed by a defender’ s system (Guerra et al., 2010). The dynamic and contentious nature of this do- main, demands a thorough analysis of the dependability and security of machine learning systems, when used in cybersecurity applications. In an adversarial environment, the accuracy of clas- siﬁcation has little signiﬁcance, if an attacker can easily ev ade detection by intelligently perturbing the input samples (Kantchelian et al., 2013). Any de- ployed classiﬁer is susceptible to probing based attacks, where an adversary uses the same channel as the client users, to gain information about the system, and then subsequently uses that information to ev ade detection (Abramson, 2015; Biggio et al., 2014a). This is seen in Fig. 1 a), where the defender starts by learning from the training data and then deploys the classiﬁer C , to pro- Pr eprint submitted to Elsevier Mar ch 24, 2017 vide services to client users. Once deployed, the model C is vulnerable to adv ersaries, who try to learn the be- havior of the defender’ s classiﬁer by submitting probes as input samples, masquerading as client users. In do- ing so, the defender’ s classiﬁer is seen only as a black box, capable of providing tacit Accept / Reject feedback on the submitted samples. An adversary , backed by the knowledge and understanding of machine learning, can use this feedback to reverse engineer the model C (as C 0 ). It can then av oid detection on future attack samples, by accordingly perturbing the input samples. It was shown recently that, deep neural networks are vulnera- ble to adversarial perturbations (P apernot et al., 2016b). A similar phenomenon was shown to a ﬀ ect a wide va- riety of classiﬁers in (Papernot et al., 2016a), where it was demonstrated that adversarial samples are transfer- able across di ﬀ erent classiﬁer f amilies. Cloud based machine learning services (such as Amazon A WS Ma- chine Learning 1 and Google Cloud Platform 2 ), which provide APIs for accessing predicti ve analytics as a ser- vice, are also vulnerable to similar black box attacks (T ram ` er et al., 2016). An example of the aforementioned adversarial en vi- ronment is illustrated in Fig. 1b), where a behavioral mouse dynamics based CAPTCHA (Completely Auto- mated Public T uring test to tell Computers and Humans Apart) system is considered. Popular examples of these systems are Google’ s reCAPTCHA 3 and the classiﬁer based system dev eloped in (DSouza, 2014). These sys- tems use mouse movement data to distinguish humans from bots and provide a conv enient way to do so, rely- ing on a simple point and click feedback, instead of re- quiring the user to infer garbled text snippets (DSouza, 2014). The illustrati ve 2D model of Fig. 1 b), shows a linear classiﬁer trained on the two features of - Mouse mov ement speed and Click time. An adversary , aim- ing to ev ade detection by this classiﬁer , starts by guess- ing the click time as a key feature (intuitiv e in this set- ting), and then proceeds to makes probes to the black box model C , to learn its behavior . Probes are made by going through the spectrum of av erage reaction times for humans 4 , guided by the Accept (green) / Reject (red) feedback from the CAPTCHA server . The information learned by reconnaissance on the black box system, can then be used to modify the attack payload so as to subse- quently ev ade detection. While, this example was sim- plistic, its purpose is to illustrate the adversarial en vi- 1 https://aws.amazon.com/machine- learning/ 2 cloud.google.com/machine- learning 3 www.google.com/recaptcha 4 www.humanbenchmark.com/tests/reactiontime (a) An adversary making probes to the black box model C , can learn it as C ’, using active learning. (b) Example task of attacking behavioral CAPTCHA. Black box model C , based on Mouse Speed and Click Time features, is used to detect benign users from bots. Adv ersary can reverse engineer C as C 0 , by guessing click time feature and making probes based on the human response time chart, using the same input channels as regular users. Figure 1: Classiﬁers in adversarial en vironment, a) shows the general adversarial nature of the problem and b) shows an example consider- ing a behavioral CAPTCHA system. ronment in which classiﬁers operate. Practical deployed classiﬁers tend to be more complex, non linear and mul- tidimensional. Howe ver , the same reasoning and ap- proach can be used to e vade complex systems. An ex- ample of this is the good words and bad words attacks on spam detection systems (Lowd and Meek, 2005b). By launching two spam emails each di ﬀ ering in only one word ’Sale’ , it can be ascertained that this word is important to the classiﬁcation, if the email containing that word is ﬂagged as spam. Kno wing this information, the adversary can modify the word to be ’Sa1e’ , which looks visually the same b ut av oids detection. These ev a- sion attacks are non-intrusiv e in nature and di ﬃ cult to eliminate by traditional encryption / security techniques, because they use the same access channels as regular input samples and they see the same black box view of the system. From the classiﬁcation perspectiv e, these at- tacks occur at test time and are aimed at increasing the 2 false negati ve rate of the model, i.e. increase the num- ber of Malicious samples classiﬁed as Le gitimate by C (Barreno et al., 2006; Biggio et al., 2013). Data driven attacks on deployed classiﬁcation sys- tems, presents a symmetric ﬂip side to the task of learn- ing from data. Instead of learning from labeled data to generate a model, the task of an attacker is to learn about the model, to generate ev asive data samples (Abram- son, 2015). With this motiv ation, we propose the Seed- Explore-Exploit(SEE) framework in this paper , to ana- lyze the attack generation process as a learning problem, from a purely data driv en perspective and without in- corporating any domain speciﬁc knowledge. Research work on detecting concept drift in data streams(Sethi et al., 2016a,b), moti vated the need for a formal anal- ysis of the vulnerabilities of machine learning, with an initial e valuation proposed in our work in (Sethi et al., 2017). In this paper , we extend the earlier version with: i) incorporating and ev aluating e ﬀ ects of Diversity of attacks on the defender’ s strategy , ii) introducing adv er- sarial metrics of attack quality and the e ﬀ ects of vary- ing the parameters of attack algorithms, iii) extensiv e detailed experimentation of the frame work using a v a- riety of defender models and the Google Cloud Predic- tion Service, and iv) e xperimentation simulating e ﬀ ects of di versity on blacklisting based countermeasures. The main contributions of this paper are: • A domain independent data driven framework is presented, to simulate attacks using an Exploration-Exploitation strategy . This generic framew ork and the algorithms presented, can be used to analyze simple probing attacks to more so- phisticated rev erse engineering attacks. • Formal adversarial model and adversary’ s metrics are proposed, as a background for sound scientiﬁc dev elopment and testing for secure learning frame- works. • Empirical analysis on 10 real world datasets, demonstrates that feature space information is suf- ﬁcient to launch attacks against classiﬁers, irre- spectiv e of the type of classiﬁer and the application domain. Additional experimentation on Google’ s Cloud Prediction API, demonstrates vulnerability of remote black box prediction services. • The analysis of div ersity and its e ﬀ ect on black- listing based countermeasures, demonstrates that such security measures (as proposed in (Kantche- lian et al., 2013)) are ine ﬀ ective when faced with rev erse engineering based attacks of high diversity . The rest of the paper is organized as follows: Sec- tion 2, presents background and related work on the security of machine learning. Section 3, presents the formal model of an adversary and the proposed Seed- Explore-Exploit(SEE) framework, for attack genera- tion. Based on the frame work, the Anchor Points at- tack and the Rev erse Engineering attack algorithms are presented in Section 3.2 and 3.3, respectively . Experi- mental ev aluation and detailed analysis is presented in Section 4. Additional discussion about div ersity of at- tacks and its importance is presented in Section 5. Con- clusion and av enues for further research are presented in Section 6. 2. Related W ork on the Security of Machine Learn- ing One of the earliest works on machine learning secu- rity was presented in (Barreno et al., 2006), where a tax- onomy of attacks was deﬁned, based on the principles of information security . The taxonomy categorized at- tacks on machine learning systems along the three axis of: Speciﬁcity , Inﬂuence and the type of Security V i- olation, as shown in T able 1. Based on the inﬂuence and the portion of the data mining process that these at- tacks a ﬀ ect, they are classiﬁed as being either Causati ve or Exploratory (Biggio et al., 2014b). Causativ e attacks a ﬀ ect the training data while Exploratory attacks a ﬀ ect the model at test time. Speciﬁcity of the attacks, refers to whether they a ﬀ ect a set of targeted samples, in order to avoid detection by perturbing them, or if the attacks are aimed towards indiscriminately a ﬀ ecting the system, with no speciﬁc pre-selected instances. Based on the type of security violation, the attacks can aim to violate integrity , by gaining unsanctioned access to the system, or can be used to launch a denial of service av ailability attack. Causativ e attacks aim to mislead training, by poi- soning the training set, so that future attacks are eas- ily ev aded (Li and Y eung, 2014; Biggio et al., 2014a). Causativ e attacks, although severe in e ﬀ ect, can be pre- vented by careful curation of the training data (Li and Chan, 2014) and by keeping the training data secure- us- ing database security measures, authentication and en- cryption. Exploratory attacks are more commonplace, less moderated and can be launched remotely without raising suspicion. These attacks a ﬀ ect the test time data, and are aimed at reducing the system’ s predictiv e per- formance (Biggio et al., 2014b). This is done by crafting the attack samples, to ev ade detection by the defender’ s model (Biggio et al., 2013; Lowd and Meek, 2005a). 3 T able 1: Cate gorization of attacks against machine learning systems Inﬂuence Causative- Attacks inﬂuence training data, to mislead learning Exploratory- Attacks a ﬀ ect test time data, to ev ade detection Speciﬁcity T ar geted- Attack a ﬀ ect onyl particular instances Indiscriminate- Attacks irrespective of instances Security violation Inte grity- Results in increased false negati ves A vailability- Denial of service attacks, due to increased errors Once a model is trained and deployed in a cybersecu- rity application, it is vulnerable to exploratory attacks. These attacks are non intrusiv e and are aimed at gaining information about the system, which is then exploited to craft ev asive samples, to circumvent the system’ s se- curity . Since, these attacks use the same channel as the client users, they are harder to detect and pre vent. T argeted-Exploratory attacks aim to modify a speciﬁc set of malicious input samples, minimally , to disguise them as legitimate. Indiscriminate attacks are more gen- eral in their goals, as the y aim to produce any sam- ple which will result in the defender’ s model to have a misclassiﬁcation. Most work on exploratory attacks are concentrated on the targeted case, considering it as a constrained form of indiscriminate attacks, with the goal of starting with a malicious sample and making minimal modiﬁcations to it, to av oid detection (Biggio et al., 2014a, 2013; Xu et al., 2016; Pastrana Portillo, 2014; Lowd and Meek, 2005b). This idea was formal- ized in (Lowd and Meek, 2005a), where the Minimal Adversarial Cost (MA C) metric, of a genre of classi- ﬁers, was introduced to denote the ease with which clas- siﬁers of a particular type can be ev aded. The hardness of evasion was giv en in terms of the number of probes needed to obtain a lo w cost e vasi ve sample. A classiﬁer was considered easy to ev ade if making a few optimal modiﬁcations to a set of samples resulted in a high ac- curacy of e vasion. W ork in (Nelson and et al., 2010) shows that linear and con vex inducing classiﬁer are all vulnerable to probing based attacks, and (Nelson et al., 2012) presents e ﬃ cient probing strategies to carry out these attacks. Particular strategies de veloped for performing ex- ploratory attacks vary based on the amount of infor - mation available to the adversary , with a broad clas- siﬁcation presented in (Alabdulmohsin et al., 2014) as: a) Evasion attacks and b) Reverse Engineering at- tacks. Ev asion attacks are used when limited informa- tion about the system is a v ailable, such as a fe w le- gitimate samples only . These legitimate samples are exploited by masking techniques such as- mimicking (Smutz and Stavrou, 2016) and spooﬁng (Akhtar et al., 2011), which masquerade malicious content within the legitimate samples. The mimicry attack was presented in (Smutz and Stavrou, 2016), where the Mimicus tool 5 was de veloped, to implement ev asion attacks on pdf documents, by hiding malicious code within benign documents. The good words attacks on spam emails uses a similar technique (Lowd and Meek, 2005b). A spam email is inserted with benign looking words, to ev ade detection. Similarly , spooﬁng attacks are com- mon in biometrics(Akhtar et al., 2011) and for phishing websites(Huh and Kim, 2011), where visual similarity can be achieved with totally di ﬀ erent content. A gen- eral purpose, domain independent technique for e v asion was presented in (Xu et al., 2016). Here, using genetic programming, variants of a set of malicious samples were generated as per a monotonically increasing ﬁt- ness function, which denoted success of ev asion. This is an attracti ve technique due to its generic approach, but limited probing b udgets and lack of a graded ﬁtness function, are some of its practical limitations. In the presence of a large probing b udget, or speciﬁc informa- tion about the defender’ s classiﬁer model, the gradient descent ev asion attack of (Biggio et al., 2013), can be used. This attacks relies on knowing the exact classi- ﬁer function used by the defender, or the ability to re- verse engineer it using a su ﬃ cient number of probes. Once information about the classiﬁer is kno wn, the at- tack uses a gradient descent strategy to ﬁnd an optimal low cost e v asion sample for the classiﬁer . Search strate- gies were de veloped for a wide range of classiﬁers with di ﬀ erentiable decision functions, including neural net- works, non-linear Support vector machines, one class classiﬁers and for classiﬁers operating in discrete fea- 5 www.github.com/srndic/mimicus/blob/master/ mimicus/attacks/mimicry.py 4 Figure 2: Gradient descent ev asion attack over 500 iterations. Left- Initial image of digit 3, Center- Image which ﬁrst gets classiﬁed as 7, Right- Image after 500 iteration.(Biggio et al., 2013) ture spaces (Biggio et al., 2013). An illustration of the gradient descent attacks for masquerading a sample is shown in Fig. 2, where the image 3 is modiﬁed to be classiﬁed as 7 ov er 500 iterations of gradient descent. Rev erse engineering the defender’ s model provides av enues for sophisticated exploratory attacks, as it ex- poses features important to the classiﬁer, to be used for mimicry attacks or lar ge scale indiscriminate attacks. Perfect re verse engineering is not needed, as an adver - sary is interested only in identifying the portion of the data space which is classiﬁed as Le gitimate . Rev erse engineering was ﬁrst employed in (Lowd and Meek, 2005a), where a sign witness test was used to see if a particular feature had a positive or negati ve impact on the decision. Rev erse engineering of a decision tree classiﬁer , as a symmetric model for defender and adver - sary , was presented in (Xu et al., 2014). (Xu et al., 2016) used genetic programming as a general purpose reverse engineering tool, under the assumption of known train- ing data distribution and feature construction algorithm. The genetic programming output, because of its intu- itiv e tree structure, was then used to guide e vasion at- tacks on intrusion detection systems. The idea of re- verse engineering was linked to that of active learning via query generation in (Alabdulmohsin et al., 2014), where the robustness of SVM to reverse engineering is tested using activ e learning techniques of random sam- pling, uncertainty sampling and selectiv e sampling. In the above mentioned works of targeted-exploratory attacks, it is assumed that if an e v asion is e xpensiv e (far from the original malicious sample), the adversary will giv e up. The above techniques are not designed for a determined adversary , who is willing to launch indis- criminate attacks. An adv ersary who wants to launch an indiscriminate attack will not bother with the near opti- mal ev asion problem (Nelson and et al., 2010). These type of attacks have been lar gely ignored, with the only mention we found was in (Zhou et al., 2012), where it is termed - the free range attack, as an adversary is free to mov e about in the data space. In such attacks, the adver- sary will ﬁrst analyze the vulnerabilities of the model, looking for prediction blind spots, before attempting an attack. Analyzing performance of models under such attack scenarios is essential to understanding its vulner- abilities in a more general real world situation, where all types of attacks are possible. Also, while most recent methodologies de velop attacks as an experimental tool to test their safety mechanisms, there is very few works (Biggio et al., 2013; Xu et al., 2014; W agner and Soto, 2002; Pastrana Portillo, 2014), which hav e attempted to study the attack generation process itself. Our proposed work analyzes Indiscriminate-Exploratory-Inte grity vi- olating attacks, under a data driv en frame work, with div erse adversarial goals and while considering only a black box model for the defender’ s classiﬁer . W e ana- lyze the attacks from an adversary’ s point of view , con- sidering the adversarial samples generation process, so as to understand the vulnerabilities of classiﬁers and to motiv ate the dev elopment of secure machine learning architectures. 3. Proposed Methodology for Simulating Data Driven Attacks on Classiﬁers Data driven exploratory attacks on classiﬁers, a ﬀ ect the test time data seen by a deployed classiﬁer . An ad- versary intending to ev ade classiﬁcation, will begin by learning information about the system, over time, and then will launch an attack campaign to meet its goals. The adversary can only interact with the system as a black box model, receiving tacit Accept / Reject feedback for the submitted samples. Ho we ver , an adversary can only make a limited number of probes, before it gets de- tected or runs out of resources. Additionally , we assume a minimal shared kno wledge en vironment between the adversary and the defender (Rndic and Laskov, 2014). Only the feature space information is shared between the two parties, as both operate on the same data space. All other information - such as the defender’ s classiﬁer type, the model parameters and the training data, is kept hidden by the defender . Both the adversary and the de- fender are assumed to be capable machine learning ex- perts, who are equipped with the tools and understand- ing of using a data driv en approach to best suit their goals. Based on this intuitive understanding, the formal model of an adversary based on it’ s knowledge, goals and resources (Biggio et al., 2014a), is presented belo w: • Knowledge- The adversary is aware of the num- ber , type and range of features, used by the classi- ﬁcation model. This could be approximated from publications, publicly a vailable case studies in re- lated applications, or by educated guessing (Rndic and Laskov, 2014). For example, in case of spam 5 classiﬁcation, the feature domain could be the dic- tionary of English words, which is publicly av ail- able and well known. This represents a symmetric learning problem with both parties operating on the same feature space. No other information about the defender’ s model is known by the adversary . • Goals- The adversary intends to cause false nega- tiv es for the defender’ s classiﬁer , on the submitted attack samples. Additionally , the adversary also wants the attacks to be robust, such that it can avoid being detected and stopped by simple blacklisting techniques (Bilge and Dumitras, 2012). From a data driv en perspective, the attacker aims to avoid detection by generating an attack set with high div ersity and variability . While, repeating a sin- gle conﬁrmed attack point, over and o ver , leads to ensured false negativ es, such attacks are easily stopped by blacklisting that single point. W e con- sider serious attackers only , who aim to force the retraining of the defender’ s classiﬁcation system. • Resources- The adversary has access to the sys- tem only as a client user . It can submit probes and receiv e binary feedback on it, upto a limited prob- ing budget, without being detected. The adversary does not have control over the training data or the model trained by the defender . The presented model of the adv ersary represents a general setting, where an attacker can take the form of an end user and then attack the system over time. Mod- ern day web applications, which aim to reach as many users as possible, all operate under this en vironment and are susceptible to data driven attacks. Based on the ad- versary’ s model, the attack can be formalized here. A classiﬁer C , trained on a set of training data D T r ain , is responsible for classifying incoming samples into Le- gitimate or Malicious classes. An adversary aims to generate an attack campaign of samples D 0 Att ack , such that C ( D 0 Att ack ) has a high false ne gati ve rate. The adv er- sary has at its disposal, a budget B E x plore of probing data D 0 E x plore , which it can use to learn C and understand it as C 0 ( D 0 E x plore ). The number of attack samples ( N Att ack ) should be much larger than B E x plore , to justify expendi- ture on the adv ersary’ s part. This speciﬁed notation will be used through the rest of the paper . The Seed-Explore-Exploit (SEE) framework is pre- sented in Section 3.1, which provides an overvie w of the attack paradigm. T wo speciﬁc attack strategies de- veloped under the SEE framework, the Anchor Points attacks (AP) and the Rev erse Engineering attacks (RE), are presented in Section 3.2 and Section 3.3, respec- tiv ely . 3.1. The Seed-Explore-Exploit (SEE) F rame work The SEE framew ork employs a data driven ap- proach for generating adversarial samples. The idea of Exploration-Exploitation is common in search based optimization techniques, where the goal is to learn the data space and then emphasize only on the promising directions (Chen et al., 2009). An adversary can also utilize a similar strategy , to best utilize the exploration budget ( B E x plore ), such that the resulting attack samples ( D 0 Att ack ) have high accuracy and high diversity . The spe- ciﬁc steps of the framew ork are explained belo w: • Seed- An attack starts with a seed phase, where it acquires a legitimate sample (and a malicious sam- ple), to form the seed set D 0 S eed . This seed sample can be acquired by random sampling in the fea- ture space, by guessing a few feature values, or from an external data source of a comparable ap- plication (Papernot et al., 2016a). For the case of a spam classiﬁcation task, picking an email from one’ s own personal inbox would be a functional legitimate seed sample. • Explore- Exploration is a reconnaissance task, starting with D 0 S eed , where the goal is to obtain maximum div erse information, to understand the cov erage and e xtent of the space of legitimately classiﬁed samples. In this phase, the adversary submits probes and receives feedback from the de- fender’ s black box. The defender can be probed upto a budget B E x plore , without being thwarted or detected. T o av oid detection, it is natural that the adversary needs to spread out the attacks over time and data space, in which case the B E x plore is the time / resources av ailable to the adversary . The ex- ploration phase results in a set of labeled samples D 0 E x plore , and the goal of the adversary is to best choose this set based on it’ s strategy . • Exploit- The information gathered in the explo- ration phase is used here to generate a set of attack samples D 0 Att ack . The e ﬃ cacy of the attack is based on the accurac y and the di versity of these samples. The SEE framework provides a generic way of deﬁn- ing attacks on classiﬁers. Speciﬁc instantiations of the three phases can be dev eloped, to suit one needs and simulation goals. 6 3.2. The Anchor P oints Attack (AP) The Anchor Points attack is suited for adversaries with a limited probing budget B E x plore , who hav e a goal of generating ev asive samples for immediate beneﬁts. An example of this would be - zero day exploits, where an adversary wants to exploit a new found vulnerabil- ity , before it is ﬁxed (Bilge and Dumitras, 2012). These attacks start by obtaining a set of samples classiﬁed as Legitimate by C , called the Anchor Points, which serve as ground truth for generating further attack samples. From a data driv en perspectiv e, this attack strategy is deﬁned under the SEE framew ork as giv en belo w . • Seed- The attack begins with a single legitimate sample as the Seed ( D 0 S eed ). • Explore- After the initial seed has been obtained (provided or randomly sampled), the exploration phase proceeds to generate the set of Anchor Points, which will enable the understanding of the space of samples classiﬁed as Le gitimate . The ex- ploration phase is described in Algorithm 1, and is a radius based incremental neighborhood search technique, around the seed samples, guided by the feedback from the black box model C . Div er- sity of search is maintained by dynamically adjust- ing the search radius ( R i ), based on the amount of ground truth obtained so f ar (Line 5). This ensures that radius of exploration increases in cases where the number of legitimate samples obtained is high, and vice versa, thereby balancing di versity of sam- ples with their accuracy . Samples are explored by perturbing an already explored legitimate sample (Seed sample in case of ﬁrst iteration), within the exploration radius (Line 7). The ﬁnal exploration dataset of Anchor Points - D 0 E x plore , is comprised of all explored samples x i , for which C ( x i ) indicated the Legitimate class label. The exploration phase is illustrated on a synthetic 2D dataset in Fig. 3, where the neighborhood radius R i indicates the ex- ploration neighborhood of a sample. • Exploit- The anchor points obtained as D 0 E x plore , forms the basis for launching the dedicated attack campaign on the classiﬁer C . The exploitation phase (Algorithm 2) combines two techniques to ensure high accuracy and diversity of attack sam- ples: a) Simple perturbation- The anchor point samples are perturbed, similar to the exploration phase, using a radius of exploitation- R E x ploit (Line 4) and, b) Conve x combination- The perturbed samples are combined using conv ex combination Algorithm 1: AP- Exploration Phase Input : Seed Data D 0 S eed , Defender black box C . P arameters : Exploration b udget B E x plore , Exploration neighborhood- [ R min , R ma x ] Output: Exploration data set D 0 E x plore 1 D 0 E x plore ← D 0 S eed 2 count legitimate = 0 3 for i = 1 .. B E x plore do 4 x i ← Select random sample from D 0 E x plore 5 R i = ( R ma x − R min ) ∗ ( count legit imat e / i ) + R min 6  Dynamic neighborhood search 7 ˆ x i ← P erturb ( x i , R i )  perturbed sample 8 if C.pr edict( ˆ x i ) is Le gitimate then 9 D 0 E x plore ∪ ˆ x i 10 count legitimate ++ 11 Procedur e Perturb(sample, R N eigh ) 12 return sample += random(mean = 0, std = R N eigh ) of samples, tw o at a time (Line 7). This is in- spired by the Synthetic Minority Oversampling T echnique (SMO TE), which is a popular o versam- pling technique for imbalanced datasets (Chawla et al., 2002). The attack set D 0 Att ack , sho wn in red in Fig. 3, is the ﬁnal attack on the classiﬁer C . The performance of the AP attack is largely depen- dent on the probes collected in the initial seed and ex- ploration phase. As such, maintaining diversity is key , as larger coverage ensures more ﬂexibility in attack gen- eration. By the nature of these attacks, they can be thwarted by blacklists capable of approximate match- ing (Prakash et al., 2010). Nev ertheless, they are suited for adhoc swift blitzkrie gs, before the defender has time to respond. 3.3. The Reverse Engineering Attac k (RE) In case of sophisticated attackers, with a large B E x plore , direct rev erse engineering of the classiﬁcation boundary is more advantageous. It pro vides a better un- derstanding of the classiﬁcation landscape, which can then be used to launch large scale e vasion or a vailability attacks (Kantchelian et al., 2013). Re verse engineering could also be an end goal in itself, as it provides in- formation about feature importance to the classiﬁcation task (Lo wd and Meek, 2005a). A re verse engineering attack, if done e ﬀ ecti vely , can avoid detection and make retraining harder on the part of the defender . Howe ver , unlike the AP attacks, these attacks are a ﬀ ected by the 7 Figure 3: Illustration of AP attacks on 2D synthetic data. (Left - Right) : The defender’ s model from it’ s training data. The Exploration phase depicting the seed(blue) and the anchor points samples(purple). The Exploitation attack phase samples (red) generated based on the anchor points. Algorithm 2: AP- Exploitation Phase Input : Exploration data set D 0 E x plore , Number of attacks N Att ack , Radius of Exploitation R E x ploit Output: Attacks set D 0 Att ack 1 D 0 Att ack ← [] 2 for i = 1 .. N Att ack do 3 x A , x B ← Select random samples from D 0 E x plore 4 ˆ x A , ˆ x B ← Per t ur b ( x A , R E x ploit ), Pert ur b ( x B , R E x ploit ) 5  Random perturbation 6 λ = rand om (0 , 1)  number in [0,1] 7 att ack sam ple i ← ˆ x A ∗ λ + (1 − λ ) ∗ ˆ x B 8  Conv ex combination 9 D 0 Att ack ∪ at t ack sam ple i 10 Procedur e Perturb(sample, R E x ploit ) 11 return sample += random(mean = 0, std = R E x ploit ) type of model used by the black box C , the dimensional- ity of the data and the number of probes av ailable. Ne v- ertheless, the goal of an adversary is not to exactly ﬁt the decision surface, but to infer it su ﬃ ciently , so as to be able to generate attacks of high accuracy and diver - sity . As such, a linear approximation to the defender’ s model and a partial rev erse engineering attempt should be su ﬃ cient for the purposes of launching a reduced ac- curacy attack. This reduction in accuracy can be com- pensated for by launching a massiv e attack campaign, exploiting the information provided by the re verse engi- neered model C 0 . E ﬀ ectiv e rev erse engineering relies on the availabil- ity of informativ e samples. As such, it is necessary to use the probing budget B E x plore e ﬀ ectiv ely . Random sampling can lead to wasted probes, with no additional information added, making it ine ﬀ ecti ve for the pur- poses of this attack. The query synthesis technique of (W ang et al., 2015), generates samples close to the clas- siﬁcation boundary and spreads the samples along the boundary , to provide a better learning opportunity . The approach of (W ang et al., 2015) was dev eloped for the purpose of activ e labeling of unlabeled data. W e mod- ify the approach to be used for rev erse engineering as part of the SEE framework, where the attacker learns a surrogate classiﬁer C 0 , based on probing the defender’ s black box C . The SEE implementation of the RE attack is giv en belo w: • Seed- The seed set consists of one legitimate and one malicious class sample. • Explore- The exploration phase (Algorithm 3) uses the Gram-Schmidt process (W ang et al., 2015) to generate orthonormal samples, near the mid- point of any two randomly selected seed points of opposite classes (Line 8). This has the e ﬀ ect of generating points close to the separating decision boundary of the two classes, and also of spreading the samples along this boundary’ s surface, as de- picted in the exploration phase of Fig. 4. The mag- nitude of the orthonormal vector is set based on λ i , which is selected as a random value in [0, λ ma x ], to impart diversity to the obtained set of samples (Line 10-11). At the end of the exploration phase, the resulting set of labeled samples ( D 0 E x plore ), is used to train a linear classiﬁer of choice, to form the surrogate rev erse engineered model C 0 (Line 19). Fig. 4 shows the rev erse engineered model 8 Algorithm 3: RE Exploration - Using Gram- Schmidt process. Input : Seed Data D 0 S eed , Defender black box model C . P arameters : Exploration budget B E x plore , Magnitude of dispersion λ ma x Output: Exploration data Set D 0 E x plore , Surrogate classiﬁer C 0 1 D 0 E x plore L = Legitimate samples of D 0 S eed 2 D 0 E x plore M = Malicious samples of D 0 S eed 3 for i = 1 .. B E x plore do 4 x L ← Select random samples from D 0 E x plore L 5 x L ← Select random samples from D 0 E x plore M 6 x 0 = x L − x M 7 Generate random vector x R 8 x R = x R − < x R , x 0 > < x 0 , x 0 > ∗ x 0 9  Gram-Schmidt process - x R orthogonal to x 0 10 λ i = rand om (0 , λ ma x ) 11 x R = λ i norm ( x R ) * x R 12  set magnitude of orthogonal midperpendicular 13 x S = x R + ( x L + x M ) / 2  Set x R to midpoint 14 if C.pr edict( x S ) is Le gitimate then 15 D 0 E x plore L ∪ x S 16 else 17 D 0 E x plore M ∪ x S 18 D 0 E x plore = D 0 E x plore L ∪ D 0 E x plore M 19 T rain C 0 using D 0 E x plore 20  Training can be based on linear classiﬁer of choice (red), as learned from the original black box clas- siﬁer C (green). • Exploit- The surrogate model C 0 , can be used to generate attacks with high accuracy and diversity . Ideally , a set of random points can be generated and veriﬁed against the rev erse engineered model C 0 , before adding them to the attack set D 0 Att ack . Howe ver , a practical and e ﬃ cient way would be to use the exploration set samples D 0 E x plore of Al- gorithm 3, as a seed set to generate a set of an- chor points as in Algorithm 1, with the exception that we probe C 0 instead of the original model C . Since C 0 is a locally trained model, probing it does not impact B E x plore . Thus allowing an adversary to make a large number of probes, at theoretically zero cost. The anchor points obtained can then be used to generate the attack samples using Algo- rithm 2. A larger attack radius R E x ploit can be used with this attack strategy , as additional validation is av ailable via the model C 0 . The RE attack strategy is suited for a patient adver - sary , who spends time / e ﬀ ort to probe the system and learn it, so as to hav e an e ﬀ ectiv e attack with high di- versity . Such attacks, are often hard to detect and stop by simple blacklisting techniques. Howe ver , the success of this attack relies on the goodness of the reverse en- gineered model, and could be a ﬀ ected by the nature of learning employed by the black box. 4. Experimental Evaluation This section presents experimental ev aluation of the AP and the RE approaches, on classiﬁers trained with 7 real world datasets. Additionally , e valuation on 3 datasets from the cybersecurity domain is presented, to demonstrate the vulnerabilities of machine learning sys- tems in adversarial milieus, to exploratory attacks. The experiments are presented from an adversary’ s point of view , who wishes to ha ve e ﬀ ectiv e attacks with high accuracy and div ersity . Section 4.1 presents the met- rics and the experimental protocol used, to encourage reproduce-ability of results. Experimental results and discussions is presented in Section 4.2 4.1. Experimental Methods and Setup 4.1.1. Adversary Metrics for Attack Quality An adversary aiming to create maximum impact, needs to make the set of attack samples D 0 Att ack - Ac- curate and Diverse . Accuracy ensures that the attack samples will cause an increase in the false negati ve rate of the defender’ s model C . While, di versity ensures that the attack set has enough variability , so that they can go unnoticed for a long time. These intuitive ideas are quantiﬁed using 4 proposed quality metrics, to mea- sure adversary e ﬀ ectiveness. The E ﬀ ectiv e Attack Rate (EAR) measures the accurac y of attacks, and the 3 met- rics: De viation of attacks ( σ E A ), K-Nearest Neighbor Distance (KNN-dist) and the Minimum Spanning T ree distance (MST -dist), collecti vely represent the diversity of attacks. The metrics are based on the following deﬁ- nition of e ﬀ ectiv e attacks (EA): E A = n x : C ( x ) = Legit imate ∧ x ∈ D 0 Att ack s o (1) Based on Eqn. 1, an attack sample is e ﬀ ective if it is classiﬁed as Le gitimate by the black box C . The adver - sarial quality metrics o ver the set EA are deﬁned belo w: 9 Figure 4: Illustration of RE attacks on 2D synthetic data. (Left - Right) : The defender’ s model based on training data. The Exploration phase depicting re verse engineering(red) using the Gram-Schmidt orthonormalization process. The Exploitation attack phase samples generated after validation from the surrogate classiﬁer (red samples). a ) E ﬀ ective Attac k Rate (EAR) : This is the accuracy of attacks, measured as the ratio of attack samples which successfully ev ade the defenders classiﬁer , giv en by Eqn. 2. A value of 1 denotes perfect ev a- sion. E AR = | E A |    D 0 Att ack s    (2) b ) Deviation of e ﬀ ective attacks ( σ E A ) : This is a measure of di versity , which computes the spread of data around its mean, giv en by Eqn. 3. σ E A = s 1 | E A − 1 | X x i  E A ( x i − µ E A ) 2 (3) where, µ E A indicates the Euclidean mean of sam- ples in the e ﬀ ective attack set EA. A large value of σ E A indicates that the data has high data space cov erage. c ) K-Nearest Neighbor distance of e ﬀ ective attacks (K N N − di st E A ) : This measure of div ersity , com- putes local density information of the data sam- ples (moti vated by (He and Carbonell, 2007)). It is computed by ﬁnding the average distance of the K-nearest neighbors of a sample, for all samples and then av eraging this value, as gi ven by Eqn. 4. K N N − d i st E A = P x  E A P K i = 1 d i st ( x , N N i ( x )) K . | E A | (4) Where, the dist(.) function computes Euclidean distance between two vectors, and N N i ( x ) gives the i th nearest neighbor of a sample x . A higher v alue of KNN-dist, indicates that data samples are rel- ativ ely far from each other and that every sample is in a locally sparse region of space, indicating higher spread. A v alue of K = 5 is chosen for ex- perimentation. d ) Minimum Spanning T r ee distance of e ﬀ ective at- tacks ( M S T − d i st E A ) : This is also a measure of di- versity , which is computed by ﬁnding the length of the minimum spanning tree over the set of EA sam- ples, as per Eqn. 5 (Lacevic and Amaldi, 2011). This is a measure which promotes ectropy or col- location of points, in an attempt to obtain a more global uniform and div erse spread of samples. This is especially useful in recognizing multiple locally dense clusters which are far from each other . M S T − d i st E A = length ( M S T ( E A )) | E A | − 1 (5) The MST measure computes cluster separation only once, as opposed to pairwise distance metrics which calculate distance between one point and e v- ery other point. Thus the MST provides a better sense of global div ersity , by allo wing sub groups of data to ha ve less diversity . A high value of MST - distance will indicate high div ersity . The three diversity metrics are a ﬀ ected by di ﬀ erent data distributions and together they provide a holistic representation of the v ariability of the attack data. Stan- dard deviation captures the overall spread of the data and is se verely a ﬀ ected by outliers. A larger spread of data results in higher de viation, as seen in Fig. 5 a) where the de viation σ = 0.228 is higher than in b), where the deviation is σ = 0.058. Although, deviation is e ﬀ ecti ve in capturing the global spread of data, it fails at capturing the local data characteristics. As seen in Fig. 5 a) and c), which hav e v ery close de viation 10 (a) σ = 0.228; KNN-dist = 0.071; MST -dist = 0.056 (b) σ = 0.058; KNN-dist = 0.018; MST -dist = 0.015 (c) σ = 0.218; KNN-dist = 0.017; MST -dist = 0.037 (d) σ = 0.226; KNN-dist = 0.017; MST -dist = 0.063 Figure 5: V alues of σ , KNN-dist and MST -dist for di ﬀ erent 2D synthetic data distributions ov er 100 test points. values ( ∆ σ = 0.002), but totally di ﬀ erent distribution of data. These di ﬀ erences are caught by the K N N − d i st metric, which is higher for scattered data (Fig. 5 a), KNN-dist = 0.071), as compared to closely packed data (Fig. 5 c), KNN-dist = 0.017). Howe ver , the KNN-dist metric does not account for disjoint clusters spread out in space, as its a local measure and is myopic in scope. This distinction is caught e ﬀ ectively by the MST -dist metric, which shows a signiﬁcant di ﬀ erence in the div er- sity v alues for Fig. 5 c) and d) ( ∆ M S T − di st = 0.026), ev en though the KNN-dist metric shows no di ﬀ erence between the two. The MST metric is suitable for at- tacks such as the Anchor Points attacks, where the at- tacks are concentrated around a few ground truth points, but the ground truth points themselves are spread out in space. The three metrics together represent the v ariabil- ity of the samples, in high dimensional spaces, where a visual examination of the data is not possible. The subscript E A is omitted in the representation of the di- versity metrics, through the rest of the paper , with the implicit understanding that these metrics are computed ov er the e ﬀ ectiv e attacks set only . 4.1.2. Description of Datasets Used Experimental e valuation is performed on 10 real world datasets, the details of which are presented in T a- ble 2. The ﬁrst 7 datasets were chosen from the UCI ma- chine learning (Lichman, 2013) repository and are pop- ularly used for classiﬁcation tasks, in literature. These datasets do not traditionally embody any security risks, but were chosen to ev aluate the vulnerability of classi- ﬁers in the di ﬀ erent data domains and distributions. The Spambase 6 (Lichman, 2013), KDD99 7 (Lichman, 2013) and the CAPTHCA (DSouza, 2014) datasets are binary 6 https://archive.ics.uci.edu/ml/datasets/Spambase 7 http://kdd.ics.uci.edu/databases/kddcup99/ kddcup99.html T able 2: Description of datasets used for experimentation of SEE framew ork Dataset #Instances #Dimensions Digits08 1500 16 Credit 1000 61 Cancer 699 10 Qsar 1055 41 Sonar 208 60 Theorem 3060 51 Diabetes 768 8 Spambase 4600 57 KDD99 494021 41 CAPTCHA 1885 26 classiﬁcation tasks, which represent 3 di ﬀ erent cyberse- curity domains that use machine learning as a core tech- nique. The Spambase dataset, contains data about spam emails (such as fraud schemes, ads, etc) and legitimate personal and work emails. The KDD99 dataset is a net- work intrusion detection dataset, to classify normal con- nections from di ﬀ erent classes of attack connections. The CAPTCHA dataset was dev eloped in (DSouza, 2014), for the task of blocking bots from human users, based on their mouse movement patterns, while solving a visual image based behavioral CAPTCHA puzzle. All datasets were pre-processed by ﬁrst reducing them to a binary class problem. The Digits dataset was reduced to ha ve samples of the digit 0 and 8 only , KDD99 was reduced to represent only two classes - at- tacks and normal. The dataset was then conv erted to contain only numerical values by transforming categori- cal and nominal features to binary v ariables. The result- ing number of features is shown in T able 2. The data was then normalized to the range of [0,1]. Instances were shu ﬄ ed to remov e any bias due to inherent con- cept drift. In all datasets, the class label 1 is taken to be the Malicious class and 0 is taken as the Legitimate 11 class, as con vention. 4.1.3. Experimental Protocol and Setup All experiments begin with a seed sample, which is obtained by random sampling in the feature space. The Anchor Points(AP) attack requires only one legitimate seed sample while the Rev erse Engineering(RE) attack requires one legitimate and one malicious sample. The seed phase concludes when the minimum required seed samples are obtained. The exploration probing bud- get B E x plore is taken as 1000 samples and the num- ber of attack samples required N Att ack is taken as 2000. For the AP attack, the neighborhood radius [ R min , R ma x ] is set at [0.1,0.5] and the exploitation radius is set at R E x ploit = 0.1. In case of the RE attack, a larger exploita- tion radius is taken as R E x ploit = 0.5, due to additional val- idation from the surrogate learned classiﬁer C 0 . E ﬀ ects of varying this radius values are also presented in the analysis. The magnitude of dispersion λ ma x is taken as 0.25, and it was found that changing this had little im- pact on the ﬁnal results. The adversary’ s rev erse engi- neered model is taken as a linear kernel SVM with a high regularization constant (c = 10). This ensures that the model is robust and does not overﬁt to the explored samples, which are limited and inadequate to general- ize over the entire space. All experimentation was per- formed using Python 2.7 8 and the scikit-learn machine learning library (Pedregosa and et al, 2011). The results presented are averaged ov er 30 runs for ev ery experi- ment. 4.2. Experimental Results and Analysis Experimental analysis is presented here, by consider- ing di ﬀ erent models for the defender’ s black box, and measuring its impact on the adversary’ s e ﬀ ectiveness. Section 4.2.1 presents the results of a symmetric case, where both the adversary and the defender have simi- lar model types (linear in this case). Results of a non symmetric setting are presented in Section 4.2.2, where we consider 4 di ﬀ erent model types for the defender , while the adv ersary , agnostic of these changes, still em- ploys a linear model. Experiments on a truly remote black box model is presented in Section 4.2.3, where we present experiments performed on Google’ s Cloud Prediction API. E ﬀ ects of parameters on the adv ersary’ s performance is presented in Section 4.2.4. 8 www.python.org 4.2.1. Experiments with Linear Defender Model Experiments in this section consider a linear model for the defender’ s classiﬁer C . A linear kernel SVM (regularization parameter, c = 1) is considered. This in- formation is not av ailable to the adversary , who is ca- pable of accessing this model only via probing upto a budget B E x plore = 1000. The results of the Seed and Exploration phase are presented in T able 3. The initial accuracy of the de- fender , as perceived by cross-validation on its training dataset before deployment, is seen in Column 2 of T a- ble 3. A high accuracy ( > 70%) is seen across all the datasets. The seed phase uses random sampling in the feature space to ﬁnd seed samples. No more than 50 samples, on average, were needed for ﬁnding seeds to start the attack process. The number of Anchor Points obtained is seen to be > 50% of B E x plore , indicating the ability to launch an AP attack on all 10 high dimensional domains. For the RE attack, the rev erse engineering ac- curacy of model C 0 is computed by ev aluating it on the original dataset, as an adhoc metric of C 0 ’ s understand- ing of the original data space and the extent of rev erse engineering. After the exploration phase, 2000 attack samples are generated in the exploitation phase. The E ﬀ ective At- tack Rate (EAR) and di versity metrics are presented in T able 4 for both the AP and the RE attacks. It is seen that an EAR of 97 . 7% in the case of AP and > 91 . 2% for the RE attacks, is obtained on av erage. This is seen ev en though the defender’ s model is percei ved to hav e a high accurac y as per T able 3. Accuracy of classi- ﬁers is of little signiﬁcance if the model can be easily ev aded. The high e ﬀ ecti ve attack rate for all 10 cases, highlight the vulnerability of classiﬁcation models and the misleading nature of accuracy , in an adversarial en- vironment, irrespecti ve of the data application domain. The high EAR of the RE attacks, indicate that partial rev erse engineering and the linear approximation of the defender’ s model surface is su ﬃ cient to launch an e ﬀ ec- tiv e attack against it. This can be seen for the KDD99 dataset, which has a re verse engineering accurac y of 55% while its EAR for the RE attack was 93%. This is because, generating a high accuracy on the training dataset is not the goal of the RE approach. It is more concerned with generating a lar ge number of div erse at- tack samples which would be classiﬁed as legitimate. This is possible even with partial rev erse engineering. While a high RE accuracy indicates a high EAR (con- sider Cancer dataset), it is not a required condition for the RE attack, making it of practical use in high dimen- sional spaces. 12 T able 3: Results of Seed and Exploration phases, with linear defender model Dataset Defender’ s Initial Accuracy Random probes to ﬁnd seed Explored Anchor Points / B E x plore Accuracy of RE model C 0 Digits08 98% 4.6 ± 2.63 0.63 ± 0.01 92% Credit 79% 3.13 ± 1.89 0.71 ± 0.01 71% Cancer 97% 42.91 ± 29.36 0.99 ± 0.01 95% Qsar 87% 49.5 ± 28.81 0.99 ± 0.01 42% Sonar 88% 24.03 ± 18.92 0.98 ± 0.01 61% Theorem 72% 4.07 ± 2.52 0.67 ± 0.02 57% Diabetes 78% 2.93 ± 1.23 0.50 ± 0.02 71% Spambase 91% 20.64 ± 12.93 0.50 ± 0.02 59% KDD99 99% 6.07 ± 4.23 0.91 ± 0.01 55% CAPTCHA 100% 7.27 ± 5.35 0.92 ± 0.01 91% The div ersity of the RE attacks is higher than the AP attacks, on all three metrics, indicating - a larger spread of attacks, lo wer collocation of points and a uniform dis- tribution in the attack space. This high di versity is ob- tained for RE, while still maintaining a reasonable high attack rate. The AP attacks, produces lower div ersity but has high attack accuracy than the RE attacks. This is because the number of explored anchor points was > 50% (T able 3), allowing a large scale AP attack to be feasible. The AP attack is therefore an attractiv e quick attack strategy in high dimensional spaces, irrespectiv e of the attack domain, application type and the model used. The e ﬀ ecti veness of the RE attack depends on the ability of the surrogate model C 0 to represent the space of Le gitimate ly classiﬁed samples by C . The re verse en- gineering task is dependent on the av ailability of enough probing budget and the comple xity of the boundary rep- resented by C . This is the cause for the higher variability in the EAR v alues for RE attacks in T able 4, as opposed to the AP attacks, where attacks are more tightly packed with the obtained anchor points, leading to lo wer vari- ability . The RE approach’ s EAR is low for the Credit, the Theorem and the Spambase datasets. In case of the Credit and Theorem dataset, the defender’ s accuracy is low , indicating a nonlinear separation / inseparability of samples. The RE accuracy approaches close to the de- fender’ s accuracy , b ut since the original model C has low accuracy , the reverse engineered model can only be so good. F or the Spambase dataset, the majority of the features follow a heavy tailed distribution as shown for Feature #5 in Fig. 6. In such distributions, random sampling in the range [0,1] on each features is not the best choice. Integrating domain information which is commonly kno wn, as in the case of text datasets ha ving heavy tails, can be beneﬁcial. Howe ver , following a do- Figure 6: Distrib ution of Feature #5 for Spambase dataset, showing a heavy tail. (Red - Malicious, Blue - Legitimate) main agnostic approach here, a 71% attack rate is still achiev ed, indicating the viability of such attacks. 4.2.2. Experiments with Non-Linear Defender Model The SEE frame work considers a black box model for C . As such, it is dev eloped as a generic data driven at- tack strategy , irrespectiv e of the defender’ s model type, training data or the model parameters. T o demon- strate the e ﬃ cac y of these attacks under a variety of defender environments, experiments with di ﬀ erent non linear black box models for C are presented here. P ar- ticularly , the follo wing defender models were ev alu- ated: K-Nearest Neighbors classiﬁer with k = 3 (kNN) (Cov er and Hart, 1967), SVM with an radial basis func- tion kernel with gamma of 0.1 (SVM-RBF) (Xiaoyan, 2003), C4.5 Decision Tree (DT)(Quinlan, 1993), and a Random Forest of 50 models (RF)(Breiman, 2001), as shown in T able 5. The attack er’ s model is kept the same as before and the experiments are repeated for each of the defender’ s model. A verage v alues of EAR ov er 30 runs are reported in T able 5. The AP approach is minimally a ﬀ ected by the choice of defender’ s model, with T able 5 showing a high EAR for all defender models. The drop in case of Spam- base, is attributed to the heavy tailed distributions as 13 T able 4: Results of accurac y and div ersity of AP and RE attacks, with linear defender model Dataset Method EAR σ KNN-dist MST -dist Digits08 AP 0.96 ± 0.01 0.23 ± 0.002 0.48 ± 0.01 0.41 ± 0.01 RE 0.93 ± 0.06 0.273 ± 0.009 0.76 ± 0.01 0.65 ± 0.04 Credit AP 0.98 ± 0.01 0.218 ± 0.001 1.19 ± 0.02 1.01 ± 0.02 RE 0.80 ± 0.15 0.265 ± 0.001 2.22 ± 0.02 1.72 ± 0.31 Cancer AP 0.99 ± 0.01 0.215 ± 0.001 0.38 ± 0.01 0.33 ± 0.01 RE 0.99 ± 0.01 0.263 ± 0.001 0.5 ± 0.01 0.45 ± 0.01 Qsar AP 1 0.216 ± 0.001 1.1 ± 0.01 0.94 ± 0.01 RE 0.99 + 0.01 0.264 ± 0.001 1.71 ± 0.01 1.64 ± 0.01 Sonar AP 0.99 ± 0.01 0.215 ± 0.001 1.37 ± 0.01 1.16 ± 0.01 RE 0.98 ± 0.01 0.265 ± 0.001 2.22 ± 0.01 2.1 ± 0.015 Theorem AP 0.97 ± 0.01 0.219 ± 0.002 1.05 ± 0.02 0.89 ± 0.02 RE 0.87 ± 0.08 0.267 ± 0.002 1.96 ± 0.02 1.64 ± 0.15 Diabetes AP 0.98 ± 0.01 0.217 ± 0.003 0.27 ± 0.01 0.23 ± 0.01 RE 0.95 ± 0.04 0.262 ± 0.001 0.36 ± 0.01 0.31 ± 0.01 Spambase AP 0.93 ± 0.01 0.233 ± 0.003 0.96 ± 0.02 0.79 ± 0.02 RE 0.71 ± 0.2 0.273 ± 0.004 2.04 ± 0.06 1.39 ± 0.4 KDD99 AP 0.99 ± 0.01 0.215 ± 0.001 1.06 ± 0.01 0.91 ± 0.01 RE 0.93 ± 0.04 0.263 ± 0.001 1.71 ± 0.01 1.53 ± 0.06 CAPTCHA AP 0.99 ± 0.01 0.215 ± 0.001 0.80 ± 0.01 0.68 ± 0.01 RE 0.97 ± 0.02 0.264 ± 0.001 1.22 ± 0.01 1.12 ± 0.03 explained in Fig. 6. In case of the decision trees, the model trained for Spambase, focuses only on a few key features to perform the classiﬁcation. Random probing attacks, space out the attack samples across dimensions, without considering their feature importance to classiﬁ- cation. This leads to skipping over the ke y features in the attack generation, making the attacks less e ﬀ ective. Howe ver , this could be compensated by performing par - tial re verse engineering and using a smaller exploitation radius. The RE results are signiﬁcantly dependent on the de- fender’ s choice of model. In case of nonlinear data sep- aration, as in the Credit and the Theorem datasets, the linear approximation is a bad choice and this is reﬂected in the low attack rate. In all other cases, the lo w at- tack rate is attributed to the over simpliﬁcation of the understanding of the models, which in case of the de- cision tree and random forest tend to be complicated in high dimensional spaces. Howe ver , in a majority of the cases it is seen that a 50% attack rate is still possible with the same linear SVM model used by the adver- sary . This makes the SEE framework generally appli- cable to attack classiﬁcation systems, without explicit assumptions about model types, application domain or the parameters of classiﬁcation. The e ﬃ cacy of these approaches, highlights the vulnerability of classiﬁers to purely data driv en attacks, requiring only feature space information. 4.2.3. Experiments with Google Cloud Prediction Ser- vice T o demonstrate the applicability of the RE and the AP techniques on real world remote black box classiﬁers, we performed experiments using the Google Cloud Pre- diction API 9 . This API provides machine learning-as- a-service, by allowing users to upload datasets to train models, and then use the trained model to perform pre- diction on new incoming samples. Google’ s Prediction API, provides a black box prediction system, as they hav e not disclosed the model type or the technique used for learning, to the best of our kno wledge. As such, this provides for an ideal test of the SEE ’ s attack models, where the defender is remote, accessed from a client and has no information about the defender’ s models (Paper - not et al., 2016a). W e use the API’ s Python client library to access the cloud service, and the results on the three cybersecurity datasets are sho wn in T able 6. The results of the experiment demonstrate that the AP and the RE attacks are e ﬀ ectiv e in attacking the de- fender’ s classiﬁer , by generating a high EAR over all datasets. The div ersity of the RE approach is seen to 9 https://cloud.google.com/prediction/ 14 T able 5: E ﬀ ecti ve Attack Rate (EAR) of AP and RE attacks, with non linear defender’ s model (Low EAR v alues are italicized.) kNN SVM-RBF DT RF Dataset AP RE AP RE AP RE AP RE Digits08 0.89 0.96 0.97 0.89 0.87 0.63 0.85 0.48 Credit 0.96 0.78 0.94 0.53 0.79 0.42 0.79 0.33 Cancer 0.99 0.99 0.99 0.99 0.97 0.89 0.99 0.98 Qsar 1 0.99 0.99 0.99 0.96 0.76 0.99 0.99 Sonar 0.99 0.98 1 1 0.97 0.62 0.99 0.95 Theorem 0.97 0.813 0.95 0.5 0.95 0.79 0.62 0.78 Diabetes 0.99 0.935 0.99 0.9 0.83 0.63 0.88 0.61 Spambase 0.93 0.99 0.48 0.84 0.08 0.11 0.99 0.98 KDD99 0.99 0.93 1 0.99 0.89 0.54 0.92 0.27 Captcha 0.99 0.92 0.99 0.92 0.97 0.83 0.93 0.89 be higher for the RE attacks on all three metrics of σ , K N N − d i st and M S T − d i st , indicating the variabil- ity of attacks achiev ed using the RE approach, in a real world setting. Furthermore, the RE accuracy in case of the Spambase dataset (48.1%) highlights that, lin- ear approximation and partial reverse engineering are su ﬃ cient to launch an e ﬀ ecti ve RE attack (EAR = 1). These experiments use the same exploration budget ( B E x plore = 1000) as the previous sections, to generate at- tacks of high accuracy and high div ersity . In a truly blind-folded setting, where we have no prior informa- tion about the defender’ s classiﬁer , a budget of 1000 ( ≈ $0.5) 10 samples indicates the relative ease with which classiﬁers can be e vaded and the need for a more com- prehensiv e defense strategy , be yond a static machine learning model. 4.2.4. E ﬀ ects of V arying B E x plore and R E x ploit In e valuating the AP and RE approaches, the R E x ploit was kept ﬁx ed at 0.1 for AP and 0.5 for RE. This was intuitiv ely moti vated, as conﬁdence in attacks would re- duce as distance from anchor points increases, as they are the only ground truth information a vailable to the at- tackers in the AP strategy . E ﬀ ect of increasing R E x ploit , on the accuracy and di versity of AP attacks, is shown in Fig. 7. The Credit, Theorem and the Spambase datasets were chosen for these e valuation, as the y ha ve low EAR for the RE approach (T able 4) and could therefore bene- ﬁt from parameter tuning. E ﬀ ect of increasing R E x ploit to increase di versity of AP attacks, and increasing B E x plore to increase EAR of RE attacks, as viable alternativ es to improv e performance over these three datasets is ana- lyzed and presented. 10 https://cloud.google.com/prediction/pricing The e ﬀ ectiv e attack rate (EAR) reduces with an in- crease in the e xploitation radius, as seen in Fig. 7a), because attack samples mov e away from the anchor points. There is an associated increase in the di ver- sity using both KNN-dist and MST -dist measures, as shown in c) and e). Comparison of div ersity and EAR at R E x ploit = 0.5 for the RE and AP approach shows, that for increasing diversity it is much better to switch to the RE approach instead of increasing R E x ploit arbitrarily , as the e ﬀ ectiveness of attacks starts dropping rapidly with increased radius. The drop in MST -dist in Fig. 7 e) is due to the reduction of the size of the E ﬀ ective Attack set (EA). As increasing div ersity for AP approach leads to a drop in EAR, we in vestigate if we can increase the EAR of the RE approach while maintaining its high di versity . Increasing the exploration budget increases the EAR, due to av ailability of labeled training data for the re- verse engineered model, leading to better learning of the data space. The increase in EAR ultimately plateaus, as per the Probably Approximate Learning(P A C) prin- ciples (Haussler, 1990), indicating that it is not neces- sary to arbitrarily keep increasing this b udget. The knee point is seen in Fig. 7b) (around 1500 for all datasets). After the knee point, the EAR of all three datasets is 85%, and adding more probing budget has little impact on the EAR or the di versity(Fig. 7 d, f). It is necessary to hav e su ﬃ cient probing b udget to reach this knee point, to allo w e ﬀ ectiv e re verse engineering in complex data spaces. This extra e ﬀ ort provides long term beneﬁts as it leads to increased diversity of attacks. RE is suitable for patient adversaries who want to apply data science in breaking the system. In case of a low budget the AP approach is more suitable, but with the RE strategy the assumption is that the adversary wants to spend time to learn the system before attempting an attack 15 T able 6: Results of AP and RE attacks using Google Cloud Prediction API as the defender’ s black box Spambase KDD99 CAPTCHA T raining Accuracy 93% 99% 100% Attack Metrics AP RE AP RE AP RE EAR 1 1 1 1 0.99 0.97 σ 0.216 0.264 0.218 0.265 0.218 0.265 KNN-dist 1.324 2.148 1.105 1.714 0.813 1.228 MST -dist 1.127 2.078 0.944 1.645 0.695 1.131 Accuracy of RE model C 0 48.1% 97.2% 100% (a) E ﬀ ect of R E x ploit on EAR (b) E ﬀ ect of R E x ploit on KNN-dist (c) E ﬀ ect of R E x ploit on MST -dist (d) E ﬀ ect of B E x plor e on EAR (e) E ﬀ ect of B E x plor e on KNN-dist (f) E ﬀ ect of B E x plor e on MST -dist Figure 7: E ﬀ ect of changing R E x ploit , for the AP attacks ( T op ), and B E x plor e , for the RE attacks ( Bottom ), on the E ﬀ ectiv e Attack Rate (EAR) and Div ersity (KNN-dist, MST -dist) 5. Why diversity is an important consideration in designing attacks? Throughout the design and ev aluation of the SEE framew ork, div ersity of attacks has been considered as an important goal for the adversary . This was intu- itiv ely motiv ated, as diversity ensures that the attacks hav e enough variability , so as to make its detection and prev ention di ﬃ cult. In this section, we quantify the ef- fects of diversity on the ability to thwart defenses, espe- cially those based on blacklisting of samples. Blacklists are ubiquitous in security applications, as an approach to ﬂag and block known malicious samples (Kantche- lian et al., 2013). Modern blacklists are implemented using approximate matching techniques, such as Local- ity Sensitive Hashing, which can detect perturbations to existing ﬂagged samples (Prakash et al., 2010). The goal of an attacker is to avoid detection by these black- lists, as they can make a lar ge number of attack samples unusable with a quick ﬁltering step. W ith high di ver - sity , it is unlikely that blacklisting a few samples will cause the attack campaign to stop. In case of a div erse attack, the defender will ha ve to resort to choosing be- tween maintaining a huge blacklist of samples, or to re- model the machine learning system, both of which are expensi ve tasks and require time. T o empirically ev aluate the e ﬀ ect of div ersity on blacklisting, a synthetic blacklisting experiment is pre- sented, which simulates the e ﬀ ect of approximate matching ﬁlters. The blacklist is maintained as a list BL , of previously seen attack samples, with an associ- ated approximation factor:  . An attack is detected if 16 (a) Percentage of attacks stopped by blacklist (b) MST -dist of AP and RE attacks Figure 8: Relationship between diversity and ability to thwart attacks by blacklisting samples. a ne w sample falls within  distance of any sample in the blacklist BL . The entire blacklisting process is simu- lated as follows: i) the attackers use the SEE frame work to generate N Att ack s attack samples which are submitted to the defender model C , ii) the defender is assumed to gain information over time about these N Att ack s samples and then proceeds to blacklist them by storing them in BL , iii) The attacker, still unaware of the blacklisting, continues to use its existing explored information (AP or RE model) to generate additionally more N Att ack s N ew attack samples. The e ﬀ ecti veness of the blacklisting process is computed as the number of e ﬀ ectiv e attacks in N Att ack s N ew , which are detected by BL . The percent- age of attac ks stopped , indicates e ﬀ ectiveness of black- lists and consequently the e ﬀ ect of div ersity . A small rate would indicate that blacklisting is not e ﬀ ective in stopping such attacks. Results of the blacklisting experiment, with an ap- proximation factor of  = 0 . 1, on the UCI datasets is shown in Fig. 8. In order to balance e ﬀ ects of approxi- mation across datasets, the approximation f actor is mul- tiplied by √ d , where d is the number of dimensions of the dataset. The  is chosen so as to balance the e ﬃ - cacy of the blacklists to its false positiv e rate. Increas- ing the approximation factor leads to an increase in the number of f alse positi ves, which causes legitimate sam- ples to be misclassiﬁed as attacks. Limiting false posi- tiv es is essential, as a blacklist which causes too man y false alarms would be impractical. In all experiments, the e xploration b udget is kept ﬁx ed at 1000, the ex- ploitation samples N Att ack = 2000 and an additional 2000 samples( N Att ack N ew ) are generated to test the blacklist- ing e ﬀ ects. From Fig. 8a), it is seen that the AP approach has a higher percentage of stopped attacks than the RE ap- proach, across all datasets. This is a result of the higher div ersity of the RE attacks as seen from the MST -dist metric in Fig. 8 b). The high div ersity of the RE at- tack causes blacklisting to be totally ine ﬀ ecti ve (0 at- tacks stopped) for 8 out of the 10 datasets. In these cases, increasing the  , as a countermeasure, to stop attacks is not a viable option, due to additional f alse alarms caused. A higher div ersity in RE would force the ree valuation of the security system, leading to re- design, feature engineering and collection of additional labeled samples. All these are time taking and expen- siv e e ﬀ orts, making the RE approach e ﬀ ective as an at- tack strategy . As such, if an attacker is sophisticated and has enough probing budget B E x plore , it can launch a div erse attack campaign, which is harder to stop by adhoc security measures. The AP attack provides for high accuracy and precise attacks, as was seen in T a- ble 4. Howe ver , a signiﬁcant portion of this attack cam- paign (45%, on average) is stopped by using a black- list capable of approximate matching. This experiment aims to highlight the e ﬀ ect of diversity and does not claim to be a concrete defense mechanism. Neverthe- less, it intends to be a motiv ation for further analysis into the e ﬃ cacy of incorporating such techniques for designing secure machine learning frameworks. Black- lists capable of heuristic matching, could empo wer clas- siﬁers (Kantchelian et al., 2013) with the ability to rec- ognize perturbed attack samples and as such continue to be e ﬀ ectiv e against anchor points attacks. In case of reverse engineering attacks, a blacklisting based ap- proach was seen to be ine ﬀ ectiv e. T aking preemptive counter-measures before deploying the classiﬁer, to suc- cessfully detect and mislead attacks, could be a promis- ing direction for dealing with these attacks (Hong and Kim, 2016). 6. Conclusion and Future W ork In this paper , an adversary’ s view of machine learn- ing based cybersecurity systems is presented. The pro- posed Seed-Explore-Exploit frame work, provides a data driv en approach to simulate probing based exploratory attacks on classiﬁers, at test time. Experimental e valua- tion on 10 real world datasets sho ws that, even models having high perceived accuracy ( > 90%), can be e ﬀ ec- tiv ely circumvented with a high ev asion rate ( > 95%). 17 These attacks assumed a black box model for the de- fender’ s classiﬁer and were carried out agnostic of the type of classiﬁer, its parameters and the training data used. Evaluation results considering 4 di ﬀ erent non- linear classiﬁer’ s and considering the Google Cloud Platform based prediction service, for the defender’ s black box, demonstrates the innate vulnerability of ma- chine learning in adversarial environments, and the mis- leading nature of accuracy in providing a false sense of security . Also, the ability to reverse engineer the de- fender’ s classiﬁer , and subsequently launch di verse at- tacks was demonstrated. Attacks with high div ersity were sho wn to be more potent, as they can av oid de- tection and thwarting, by countermeasures employing blacklists and approximate matching. The purpose of this work is to make the model de- signers aw are of the nature of attacks that in v ade classi- ﬁcation systems, from a purely data driv en perspectiv e. W earing the white hat, we draw attention to the vul- nerabilities introduced by using classiﬁers in cyberse- curity systems. As Sun Tzu says in The Art of W ar- ’T o know your Enemy , you must become your Enemy’(Tzu, 1963). W e hope that this work serves as background and moti vation for the de velopment and testing of no vel machine learning based security frame works and met- rics. Use of mo ving tar get defense strategies (Hong and Kim, 2016) and dynamic adversarial drift handling tech- niques (Kantchelian et al., 2013; Sethi et al., 2016a), are promising directions warranting further research. Fu- ture work will concentrate on dev eloping and analyzing preemptiv e strategies, in the training phase of the classi- ﬁers, to facilitate reliable attack detection and relearning for e ﬀ ectiv e recov ery . References References Abramson, M., 2015. T oward adversarial online learning and the sci- ence of deceptive machines. In: 2015 AAAI Fall Symposium Se- ries. Akhtar , Z., Biggio, B., Fumera, G., Marcialis, G. L., 2011. Robust- ness of multi-modal biometric systems under realistic spoof attacks against all traits. In: BIOMS 2011. IEEE, pp. 1–6. Alabdulmohsin, I. M., Gao, X., Zhang, X., 2014. Adding robustness to support vector machines against adversarial rev erse engineering. In: Proceedings of 23rd ACM CIKM. A CM, pp. 231–240. Barreno, M., Nelson, B., Sears, R., Joseph, A. D., T ygar , J. D., 2006. Can machine learning be secure? In: Proceedings of the 2006 A CM Symposium on Information, computer and communications security . ACM, pp. 16–25. Biggio, B., Corona, I., Maiorca, D., Nelson, B., ˇ Srndi ´ c, N., Lask ov , P ., Giacinto, G., Roli, F ., 2013. Evasion attacks against machine learn- ing at test time. In: Machine Learning and Knowledge Discovery in Databases. Springer , pp. 387–402. Biggio, B., Fumera, G., Roli, F ., 2014a. Pattern recognition systems under attack: Design issues and research challenges. International Journal of Pattern Recognition and Artiﬁcial Intelligence 28 (07), 1460002. Biggio, B., Fumera, G., Roli, F ., 2014b. Security e valuation of pattern classiﬁers under attack. IEEE transactions on knowledge and data engineering 26 (4), 984–996. Bilge, L., Dumitras, T ., 2012. Before we knew it: an empirical study of zero-day attacks in the real world. In: Proceedings of the 2012 A CM conference on Computer and communications security . A CM, pp. 833–844. Breiman, L., 2001. Random forests. Machine learning 45 (1), 5–32. Chawla, N. V ., Bowyer , K. W ., Hall, L. O., Ke gelmeyer , W . P ., 2002. Smote: synthetic minority over-sampling technique. Journal of ar- tiﬁcial intelligence research 16, 321–357. Chen, J., Xin, B., Peng, Z., Dou, L., Zhang, J., 2009. Opti- mal contraction theorem for exploration–exploitation tradeo ﬀ in search and optimization. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 39 (3), 680–691. Cover , T . M., Hart, P . E., 1967. Nearest neighbor pattern classiﬁcation. Information Theory , IEEE Transactions on 13 (1), 21–27. DSouza, D. F ., 2014. A v atar captcha: telling computers and humans apart via face classiﬁcation and mouse dynamics. Electronic The- ses and Dissertations-1715. Group, B. D. W ., et al., 2013. Big data analytics for security intelli- gence. Cloud Security Alliance. Guerra, P . H. C., , et al., 2010. Exploring the spam arms race to char- acterize spam ev olution. In: Proceedings of the 7th Collaboration, Electronic messaging, Anti-Abuse and Spam Conference (CEAS). Citeseer . Haussler , D., 1990. Probably approximately correct learning. Univer - sity of California, Santa Cruz, Computer Research Laboratory . He, J., Carbonell, J. G., 2007. Nearest-neighbor -based active learning for rare category detection. In: Advances in neural information processing systems. pp. 633–640. Hong, J. B., Kim, D. S., 2016. Assessing the e ﬀ ectiveness of mov- ing target defenses using security models. IEEE Transactions on Dependable and Secure Computing 13 (2), 163–177. Huh, J. H., Kim, H., 2011. Phishing detection with popular search engines: Simple and e ﬀ ective. In: Foundations and Practice of Se- curity . Springer, pp. 194–207. Kantchelian, A., Afroz, S., Huang, L., Islam, A. C., Miller , B., Tschantz, M. C., Greenstadt, R., Joseph, A. D., T ygar, J., 2013. Approaches to adversarial drift. In: Proceedings of the 2013 ACM workshop on Artiﬁcial intelligence and security . ACM, pp. 99– 110. Lacevic, B., Amaldi, E., 2011. Ectrop y of di versity measures for pop- ulations in euclidean space. Information Sciences 181 (11), 2316– 2339. Li, H., Chan, P . P ., 2014. An improved reject on negati ve impact de- fense. In: International Conference on Machine Learning and Cy- bernetics. Springer , pp. 452–459. Li, Y ., Y eung, D. S., 2014. A causativ e attack against semi-supervised learning. In: Machine Learning and Cybernetics. Springer, pp. 196–203. Lichman, M., 2013. UCI machine learning repository . URL http://archive.ics.uci.edu/ml Lowd, D., Meek, C., 2005a. Adversarial learning. In: Proceedings of the 11th A CM SIGKDD. A CM, pp. 641–647. Lowd, D., Meek, C., 2005b. Good word attacks on statistical spam ﬁlters. In: CEAS. Nelson, B., , et al., 2012. Query strategies for evading conv ex- inducing classiﬁers. The Journal of Machine Learning Research 13 (1), 1293–1332. Nelson, B., et al., 2010. Near-optimal ev asion of conve x-inducing 18 classiﬁers. arXiv preprint arXi v:1003.2751. Papernot, N., McDaniel, P ., Goodfellow , I., 2016a. Transferability in machine learning: from phenomena to black-box attacks using ad- versarial samples. arXi v preprint Papernot, N., McDaniel, P ., Jha, S., Fredrikson, M., Celik, Z. B., Swami, A., 2016b. The limitations of deep learning in adversar- ial settings. In: 2016 IEEE European Symposium on Security and Priv acy (EuroS&P). IEEE, pp. 372–387. Pastrana Portillo, S., 2014. Attacks against intrusion detection net- works: ev asion, rev erse engineering and optimal countermeasures. Doctoral Theses, Univ ersidad Carlos III de Madrid. Pedregosa, F ., et al, 2011. Scikit-learn: Machine learning in Python 12, 2825–2830. Prakash, P ., Kumar, M., Kompella, R. R., Gupta, M., 2010. Phishnet: predictiv e blacklisting to detect phishing attacks. In: INFOCOM, 2010 Proceedings IEEE. IEEE, pp. 1–5. Quinlan, J. R., 1993. C4. 5: Programming for machine learning. Mor - gan Kau ﬀ mann. Rndic, N., Laskov , P ., 2014. Practical e v asion of a learning-based clas- siﬁer: A case study . In: Security and Privac y (SP), 2014 IEEE Symposium on. IEEE, pp. 197–211. Saha, A., Sanyal, S., 2014. Application layer intrusion detection with combination of explicit-rule-based and machine learning algo- rithms and deployment in cyber-defence program. arXiv preprint Sethi, T . S., Kantardzic, M., Arabmakki, E., 2016a. Monitoring clas- siﬁcation blindspots to detect drifts from unlabeled data. In: 17th IEEE International Conference on Information Reuse and Integra- tion (IRI). IEEE. Sethi, T . S., Kantardzic, M., Hu, H., 2016b. A grid density based framew ork for classifying streaming data in the presence of con- cept drift. Journal of Intelligent Information Systems 46 (1), 179– 211. Sethi, T . S., Kantardzic, M., Ryu, J. W ., 2017. Security theater: On the vulnerability of classiﬁers to exploratory attacks. In: (Under Con- sideration) 12th Paciﬁc Asia W orkshop on Intelligence and Secu- rity Informatics. Springer . Smutz, C., Stavrou, A., 2016. When a tree falls: Using diversity in ensemble classiﬁers to identify evasion in malware detectors. In: NDSS Symposium. T ram ` er , F ., Zhang, F ., Juels, A., Reiter , M. K., Ristenpart, T ., 2016. Stealing machine learning models via prediction apis. arXiv preprint Tzu, S., 1963. The art of war . edited by samuel b. gri ﬃ th. W agner , D., Soto, P ., 2002. Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th A CM Conference on Computer and Communications Security . ACM, pp. 255–264. W algampaya, C., Kantardzic, M., 2011. Cracking the smart clickbot. In: 13th IEEE International Symposium on W eb Systems Evolu- tion (WSE),. IEEE, pp. 125–134. W ang, L., Hu, X., Y uan, B., Lu, J. g., 2015. Active learning via query synthesis and nearest neighbour search. Neurocomputing 147, 426–434. Xiaoyan, W . P . Z., 2003. Model selection of svm with rbf kernel and its application. Computer Engineering and Applications 24, 021. Xu, L., Zhan, Z., Xu, S., Y e, K., 2014. An evasion and counter-ev asion study in malicious websites detection. In: Communications and Network Security (CNS), 2014 IEEE Conference on. IEEE, pp. 265–273. Xu, W ., Qi, Y ., Evans, D., 2016. Automatically ev ading classiﬁers. In: Proceedings of the Network and Distributed Systems Symposium. Zamani, M., Movahedi, M., 2013. Machine learning techniques for intrusion detection. arXiv preprint arXi v:1312.2177. Zhou, Y ., Kantarcioglu, M., Thuraisingham, B., Xi, B., 2012. Adver- sarial support vector machine learning. In: Proceedings of the 18th A CM SIGKDD international conference on Knowledge discovery and data mining. A CM, pp. 1059–1067. ˇ Zliobait ˙ e, I., 2010. Learning under concept drift: an overview . arXiv preprint 19

Data Driven Exploratory Attacks on Black Box Classifiers in Adversarial Domains

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment