Lower Bounds for Adversarially Robust PAC Learning

Lower Bounds f or Adversarially Rob ust P A C Learning Dimitrios I. Diochnos ∗ University of V ir ginia diochnos@vir ginia.edu Saeed Mahloujifar ∗ University of V ir ginia saeed@virgin ia.edu Mohammad Mahmoody † University of V ir ginia mohammad@vir ginia.edu Abstract In this work, we in itiate a f ormal study of probably approximately co rrect (P A C) learning under ev asion attacks, where the adversary’ s goal is to miscla ssify the adversarially perturbed sample po int e x , i.e., h ( e x ) 6 = c ( e x ) , where c is the ground truth concept an d h is the learned hypothesis. Previous work on P A C learning of adversarial examples have all mo deled adversarial examples a s corrupted in puts in which the goal o f the adversary is to achieve h ( e x ) 6 = c ( x ) , wh ere x is the orig inal untamper ed instance. These two definitions of adversarial risk coin cide fo r many natural distributions, such as im ages, but ar e incompar able in gen eral. W e first prove that for m any theo retically natural input spaces of high dim ension n (e.g., isotrop ic Gau ssian in dimension n under ℓ 2 perturb ations), if the a dversary is allowed to apply up to a sublinear o ( k x k ) amo unt of perturb ations on the test instances, P AC lear ning r equires sample complexity that is exponential in n . This is in contrast with results proved using the corrupted- input f ramework, in which the sample comp lexity of robust learning is only poly nomially more. W e then for malize hybrid a ttac k s in which th e ev asion attack is p receded by a poisoning attack. This is p erhaps reminiscent o f “trapdoor attacks” in which a poisoning ph ase is inv olved as well, but the ev asion ph ase here uses the e r ror- region definition of r isk th at aims at misclassifying the perturbed instances. In this case, we sho w P AC learning is som etimes impossible all to gether, even when it is possible witho u t th e attack (e.g ., due to the bound e d V C dimension). Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Defining Adversarially Robust P A C Learning . . . . . . . . . . . . . 4 3 Lower Bounds f or P A C Learning under Evasion and H y brid Attacks . . . . 5 4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . 8 5 Conclusion and Open Questions . . . . . . . . . . . . . . . . . 9 ∗ Authors have contributed equally . † Supported by NSF CAREE R aw ard CCF- 135093 9 and Uni versity of V irginia’ s SEAS Research Innov ation A ward. 1 Intr oduction Learning pred ictors is th e task of outp utting a hypothesis h u sing a training set S in such a way that h can predic t the corre c t lab el c ( x ) of unseen instances such as x with h igh p robab ility . A normal successful learner, howe ver , co uld be vuln erable to adversarial perturbations. In particular , it was shown (Szegedy et al., 2014; Biggio et al., 2013; Good f ellow et al., 20 15) that dee p neur a l nets (DNNs) ar e vulnerab le to so called adversarial examples that are the result of small (ev en imperceptib le to hu man eyes) p erturbatio ns on the origin al in put x . Since the introd uction of such attacks, many works h av e studie d def e nses against them and mo r e attacks are introdu ced a fterwards (Biggio et al., 201 3, 20 14; Goo dfellow et al., 2015; Papernot e t al., 20 1 6b; Carlin i & W agner, 201 7; Xu et al., 20 1 7; Madry et al., 2017). A fu n damental question in robust learning is whethe r one can design learnin g algor ithms that ach ieve “generalizatio n” even unde r such adversar ial p erturbatio n s. Namely , we want to know wh en we can learn a robust cla ssifier h that still co rrectly classifies its inputs e ven if they ar e adversarially perturb ed in a limited way . Indeed , one can ask wh en the ( ε, δ ) P A C (pr obably app roximately correct) learning (V aliant, 198 4) is possible in adversarial setting s. More form ally , the goal here is to learn a robust h fr om the data set S co nsisting of m indepen dently sampled labeled (non - adversarial) instances in such a way that, with pro bability 1 − δ over the learning p r ocess, the produced h has error at most ε even under “limited” ad versarial perturbation s of the input. This limitation is carefu lly defined by some metr ic d defined over the inp ut space X and some up per b ound “b udg et” b on the amount of pertu rbations that the adversary can in troduce . I.e ., we would like to minimize AdvRisk ( h ) = Pr x ← D [ ∃ e x : d ( x, e x ) ≤ b , h ( e x ) 6 = c ( e x )] ≤ ε where AdvRisk is th e “a d versarial” risk, and c ( · ) is the gro und truth (i.e., the concep t func tio n). Error -region adversarial risk. Th e above n otion of adversarial risk h as been u sed imp licitly or explicitly in previous work (Gilmer et al., 2 018; Dioch nos et al., 201 8; Bubeck et al., 2018 a; Degwekar & V aikuntanath an , 20 19; Ford et al., 20 19) and was f ormalized by Dio chnos et al. ( 2 018) as the “error-region” adversarial risk, be c ause adversary’ s goal h ere is to push e x into the erro r region E = { x | h ( x ) 6 = c ( x ) } . Corrupted-input adversarial risk. Another notion of ad versar ial risk (tha t is similar, but still different fr om the er ror-region adversarial risk explain ed above) has b een used in many works such as (Feig e et al., 2015; M adry et al., 20 1 7) in which the pe rturbed e x is interpreted as a “cor rupted input”. Namely , h e r e the goal of the learn e r is to find the label of the o riginal untampered point x by only having its corru pted version e x , and thus ad versary’ s success criterio n is to reach d ( x, e x ) ≤ b, h ( e x ) 6 = c ( x ) . Hence, in that setting, the go a l of th e learner is to find an h that minimizes Pr x ← D [ ∃ e x : d ( x, e x ) ≤ b, h ( e x ) 6 = c ( x )] . It is easy to see that, if the ground truth c ( x ) does not ch ange und e r b - perturb ations, c ( x ) = c ( e x ) , the two notions of err or-region and c o rrupted -input ad versarial risk will be equal. In particular, this is the ca se for practical distributions o f intere st, su c h as images o r voice, where sufficiently- small pertu rbations u sually d o not chan ge h uman’ s judg m ent ab out the true label. Howe ver , if b -pertu r bations can chan ge the ground truth, c ( x ) 6 = c ( e x ) , the two definition s are incompara b le. Sev eral works ha ve already studied P A C learning with provable guaran tees u nder adversarial perturb ations (Bubeck et al., 2018b; C ullina e t al., 2018; Feige et al., 201 8; Attias et al., 201 8; Khim & Loh, 2 018; Y in et al., 2 018; Mon tasser et al., 2019). Howe ver , all th ese works use the corrupted-in put notio n of ad versarial risk. In par ticular, it is proved by Attias et al. (201 8 ) th at robust learning m ig ht require m ore da ta, but it was also sh own by Attias e t al. (2 018); Bub e c k e t al. (2018 b ) that in natural settings, if robust c la ssification is feasible, ro bust classifiers could be found with a sample comp lexity tha t is o nly polyno mially larger than that o f no rmal learning . Th is leads us to the ou r c e ntral question: What pr oblems are P AC learnable under evasion atta cks that pertu rb in stances into the err or r e gion? If P AC learnable, wha t is their sample comp lexity? 2 1.1 Our Contribution In this work, we initiate a formal stud y of P A C learning under ad versarial pertu rbations, where th e goal of the adversary is to increase the error-region adversarial risk using small (sub linear o ( k x k ) ) perturb ations of the inp uts x . Th erefore , in what fo llows, whenever w e ref er to adversarial risk, by default it means the error-region variant. Result 1: ex ponential lower bound on sample co mplexity . Supp ose the instances of a lear n ing problem come from a metric prob ability space ( X , D , d ) wh ere D is a d istribution and d is a metric defining some norm k·k . Suppose the input instances have norm s k x k ≈ n where n is a parameter related (or in fact equ al) to the data dimension . One natural setting of study fo r P A C learning is to study attackers that can only pertu rb x by a sublinear amount o ( k x k ) = o ( n ) (e.g., √ n ). Our fir st result is to prove a strong lower b ound for the samp le complexity o f P A C learnin g in this setting. W e prove that fo r many theoretically natural input spaces o f high dimension n (e.g., isotropic Gaussian in dimen sion n un der ℓ 2 perturb ations), P AC learning of certain prob lems u nder sublinear per turbation s of the test instances requir es e xpon entially many samples in n , even though the problem in the no -attack setting is P A C learnable using po ly nomially many samples. This holds e.g., when we want to learn half spaces in dimension n und e r such distributions (which is p ossible in the no-attack setting). W e note th at e ven thou gh P A C learning is defined for all d istributions, proving such lo wer bound for a specific input distribution D ov er X o nly makes the negative result str onger . Our lower bou nd is in con tr ast with previously p roved results (Attias et al., 20 18; Bub eck et al., 2018b; Monta sser et al., 2019; Cullina et al., 2 018) in which th e gap between the sample comp lexity of th e no rmal and robust learning is o nly po lynomial . Ho wever , as mentio ned before , all th e se previous resu lts are proved using the corrup ted-inpu t variant of adversarial risk. Our result extend s to any learning problem where inpu t space X , the me tric d and th e distribution D defined over them, and the class o f concept functio ns C have the following two co nditions. 1. The in puts X under th e distribution D an d small pertu r bations m easured b y the metric d for m s a co ncentrated m e tric probab ility space (Ledo ux, 2001; Milman & Sche chtman, 1986). A concen trated space h as the property that relati vely small events (e.g., o f measur e 0 . 1 ) under small (e.g., smaller than the diameter of the space) p erturbatio ns expand to cover almost all measur e ≈ 1 of the inp ut space. 2. The set of concept f u nctions C are complex enou gh to allo w pr oving lower bound s for the sample comp lexity for (distribution-de p endent) P A C learner s in the no-a tta ck setting un- der the same distribution D . Distribution-dep endent sample complexity lower b ound s are known for certain settings (Long, 1 9 95; Balcan & Long, 2 0 13; Sabato et al., 2013), how- ev er, we use a more relaxed condition that can b e applied to bro ader settings. In particu lar , we r equire that for a sufficiently small ε , there ar e two concept fun ctions c 1 , c 2 that are equal for 1 − ε fraction of inpu ts samp led from D (see Definitio n 3.3). Having the above two conditions, o ur pr oof pro ceeds as follows (I) W e show that the (no rmal) risk Risk ( h ) of a h ypoth esis prod u ced b y any learning algorith m with sub- exponential samp le co m plexity cannot be as large as an inv erse p olyno m ial over th e dimension. (II ) W e then use ideas from the works (e.g., see (Ma h loujifar et al., 2018 b )) to show that su ch sufficiently large risk will expand into a large adversarial risk of almost all inpu ts, du e to the measure con centration the input space. Remark 1.1 (Approxim a tio n error in error-region robust learning ) . If a learning problem is realiz- able in the no -attack setting, i.e., there is a hypo th esis h that has risk zero over the test instances, it means that the same hy pothesis h will hav e adversarial (tru e ) risk zero over the test instance s as well, because any pertu rbed poin t is still go ing to be correctly classified. This is in contrast with corrupted - input n otion of adversarial risk that even in r ealizable problems, th e sma llest corr u pted-in p ut (tru e) adversarial r isk could still be large, and e ven at od ds with co rrectness ( Tsipras et al., 2018). This means that o u r results rule out (efficient) P A C lea r ning ev en in the agno stic setting as well, becau se in the realizable setting there is at least o n e hypoth esis with error-region adversarial risk zero while (as we prove), in some settings le a r ning a model with adversarial risk (u nder sublinear perturb ations) close to zero requ ires exponentially many samples. Result 2: ruling o ut P A C learning under hybrid a t tacks. W e th e n study P AC lea rning under adversarial pertu rbations that happen during bo th train ing and testing phases. W e f ormalize hy - 3 brid attacks in which the final ev asion attack is preceded by a poisoning attack (Bigg io et al., 2012; Papernot et al., 2 016a). This attack model bears similar ities to “trapd oor attack s” (Gu et al., 2 017) in which a poisoning ph a se is inv olved before the e vasion attack , and here we gi ve a f ormal definitio n for P AC learn in g under suc h attacks. Ou r definition of hybrid attac k s is general and can incorp orate any n o tion o f adversarial risk, but ou r r esults fo r h ybrid attacks use the err or-r e gion adversarial risk. Under hybrid attacks, we show that P AC lea r ning is sometimes impossible all together , even though it is possible without suc h attacks. F or example, e ven if the VC dimen sion of the concept class is bound ed by n , if th e adversary is a llowed to po ison only 1 /n 10 fraction of the m trainin g examples, then it can do so in such a way that a subsequent ev asion attack could then in crease the adversarial risk to ≈ 1 . Th is means that P A C learning is in fact imp o ssible u nder such hybr id attacks. W e also no te th at classical results abo ut malicio us n oise (V aliant, 198 5; Kearns & Li, 1993) and nasty noise (Bshouty et al., 20 02) could be interpreted as ruling out P A C learning under poisonin g attacks. Ho we ver , the r e are two differences: (I) The adversary in th e se previous works nee d s to change a co nstant fraction of the training example s, while our attacker chan ges on ly a n a rbitrarily small in verse poly nomial fr action of th em. (II) Our poisoning attacker on ly removes a fraction of the training set, a nd hence it does not add any misclassified examples to the pool. Thu s the poisoning attack used here is a clean /correct label attack (Mah loujifar et al. , 2 018a; Shafahi et al., 2018). 2 Defining Adversarially Robust P A C Learning Notatio n. By e O ( f ( n )) we r efer to th e set of all func tions of the form O ( f ( n ) log ( f ( n )) O (1) ) . W e use capital calligrap hic letters (e.g., D ) for sets and capital non- c alligraphic letters (e.g., D ) for distributions. x ← D den otes sampling x from D . For an e vent S , we let D ( S ) = Pr x ← D [ x ∈ S ] . A classification pr oblem P = ( X , Y , C , D , H ) is specified by the following comp onents. The set X is the set of possible instances , Y is the set of possible la bels , D is a class of distrib utions over instances X . In the standard setting o f P A C learn ing, D includ es all d istributions, but since we deal with ne gative results, we sometim es work with fixed D = { D } d istributions, and show that ev en distrib ution-d epende n t r obust P AC learning is sometimes hard. In that case, we repr esent the problem as P = ( X , Y , C , D , H ) . The set C ⊆ Y X is th e con cept class an d H ⊆ Y X is th e hypothe sis class . In gener al, we can allo w randomized concept and hypothesis fun ctions to model, in order, label uncerta in ly ( u sually modeled by a join t d istribution over instances and labels) an d random ized prediction s. All of our r esults extend to r andom ized learners and rand omized hyp othesis function s, but for simplicity of presentation, we treat them as determ inistic mapping s. By default, we co nsider 0 -1 lo ss functions wh ere ℓoss ( y ′ , y ) = 1 [ y ′ = y ] . For a giv en d istribution D ∈ D and a concep t function c ∈ C , th e risk o f a hyp othesis h ∈ H is the expected loss of h with resp e ct to D , namely Risk ( D , c, h ) = Pr x ← D [ ℓoss ( h ( x ) , c ( x ))] . An example z is a pair z = ( x, y ) where x ∈ X and y ∈ Y . An example is usually sampled by first samplin g x ← D for some D ∈ D followed by letting y = c ( x ) for some c ∈ C . A sample seque n ce S = ( z 1 , . . . , z m ) is a sequence of m e xamp les. As is usu al, sometime s we might re f er to a sample sequence as the training set . By S ← ( D , c ( D )) m we denote the process of obtaining S by sampling m iid samples from D and labeling them by c . Our learning pro blems P n = ( X n , Y n , C n , D n , H n ) are usually par a m eterized b y n where n denotes the “data dimension” or (closely) capture the bit len gth of the instances. Th us, the “ef ficiency” of the algorithms could d e pend o n n . E ven in this case, for simplicity of nota tio n, we might simp ly write P = ( X , Y , C , D , H ) . By default, we will have C ⊆ H , in wh ich case we call P realizable . This means that fo r any training set for c ∈ C , D ∈ D , ther e is a hyp othesis that has empirical and true risk zero; tho ugh fin ding such h mig ht b e challengin g . Evasion atta cks. An e vasion attacker A is one that chan ges the test instance x , denoted as e x ← A ( x ) . The behavior and actio ns taken by A could , in general, depend o n the choices of D ∈ D , c ∈ C , and h ∈ H . As a resu lt, in ou r notation, we provid e A with ac c e ss to D , c, h by g iving them as spe c ia l inputs to A , 3 denoting the process as e x ← A [ D, c, h ]( x ) . W e use calligrap h ic fo nt A to denote a class/set of attack s. For example, A co uld contain all attackers wh o could change test instance x by at most b pertu rbations under a metric defined over X . 3 This dependence is information theoretic, and for example, A might want to find e x that is misclassified, i n which case its success is defined as h ( e x ) 6 = c ( e x ) which depends on both h, c . 4 Poisoning att acks. A poison ing attacker A is one that changes th e training sequence as e S ← A ( S ) . Such attacks, in genera l, might add examples to S , rem ove examples from S , or do bo th . The behavior and ac tions taken b y A could, in general, d epend on th e c hoices of D ∈ D , c ∈ C (but not on h ∈ H , as it is not produ c e d by the lear ner at the time of the poisonin g attack) 4 . As a result, we provide imp licit access to D, c by giving th em as special inputs to A , deno ting the p r ocess as e S ← A [ D, c ]( S ) . W e use calligraphic font A to denote a cla ss/set o f attacks. For examp le, A could contain attacks that chang e 1 /n fractio n of S on ly using clean labe ls (Mahlo ujifar et al., 2 018b; Shafahi et al., 2018). Hybrid attacks. A hyb r id attack A = ( A 1 , A 2 ) is a two phase attack in which A 1 is a poisoning attacker and A 2 is an ev asion attacker . One subtle point is that A 2 is also aware of the in ternal state of A 1 , as they are a pair of coo rdinating attack s. More form ally , A 1 outputs an extra “state” in formatio n st wh ich will be g iven as an extra inpu t to A 2 . A s discussed above, A 1 can depend on D, c , and A 2 can depen d on D , c, h as defined for ev asion and poison ing attack s. W e now define P A C lear ning und er ad versarial perturb a tio n attac k s. T o d o so, we need to first defin e our notion of ad versarial risk. W e will do so by emp loyin g the err or -r e gion notion adversarial risk as form a lize d in Dio chnos et al. (20 18) ad versary aims to misclassify th e perturbed in stan ce e x . Definition 2.1 ( Error-region (adversarial) risk) . Suppose A is an ev asion adversary and let D , c, h be fixed. The err or -r e gion ( a dversarial) risk is d efined as follows. AdvRisk A ( D , c, h ) = Pr x ← D, e x ← A [ D,c,h ]( x ) [ h ( e x ) 6 = c ( e x )] . For randomized h , the above pr obability is also over the rand omness of h chosen after e x is selected . W e now define P AC learning un der hyb rid attacks, fro m w h ich one can derive also the definition of P AC learn ing under ev asion a ttac k s and under poiso n ing attack s. Definition 2.2 ( P AC learn ing under hybrid attacks) . Suppose P n = ( X n , Y n , C n , D n , H n ) is a real- izable classification pro blem, and suppo se A is a class of h ybrid a ttacks for P n . P n is P AC learnable with sample complexity m ( ε , δ, n ) u nder hyb rid attacks of A , if th ere is a learning algorith m L such that for every n , 0 < ε, δ < 1 , c ∈ C , D ∈ D , and ( A 1 , A 2 ) ∈ A , if m = m ( ε, δ, n ) , then Pr S ← ( D ,c ( D )) m , ( e S , st ) ← A 1 [ D, c ]( S ) , h ← L ( e S )  AdvRisk A 2 [ D, c,h, st ] ( h, c, D ) > ε  ≤ δ. P AC learning u nder (pu re) poisonin g attacks or ev asion attacks could be derived fro m Definition 2 .2 by letting either o f A 1 or A 2 be a trivial attack that does no tampe r ing at all. W e also note that o ne can obtain other d efinitions of P AC learning u nder evasi on o r hyb rid at- tacks in Definition 2.2 by using other for ms o f adversarial risk, e.g., corru pted-inp ut adversarial risk (Feige et al., 20 15, 2018; Madry et al., 201 7; Schmidt et al., 201 8; Attias et al., 2018) 3 Lower Bounds for P A C Lear ning under Ev asion and Hybrid Attacks Before proving o ur main results, we need to recall the notion o f Normal Lévy families, and define a desired and comm o n pr o perty of set of conce p t f u nctions with respect to th e d istribution of in p uts. Notatio n. Let ( X , d ) be a metr ic space. For S ⊆ X , by d ( x, S ) = inf { d ( x, y ) | y ∈ S } we deno te the distance of a point x from S . W e also let S b = { y | d ( x, y ) ≤ b, x ∈ S } be the b -expansion of S . When there is also a measure D defined over the metric space ( X , d ) , the co ncentration functio n is defined and den oted as α ( b ) = 1 − inf { Pr D [ E b ] | Pr D [ E ] ≥ 1 / 2 } . Definition 3.1 (Normal Lévy families) . A family of m etric proba b ility spac es ( X n , d n , D n ) i ∈ N with concentr a tion fu n ction α n ( · ) is called a n ormal Lévy family if ther e are k 1 , k 2 , such that 5 α n ( b ) ≤ k 1 · e − k 2 · b 2 /n 4 For ex ample, an attack model might require A to choose it s perturbed instances still using corr ect/clean labels, in which case t he attack is restricted based on the choice of c ). 5 Another common formulation of Normal Lévy families uses α n ( b ) ≤ k 1 · e − k 2 · b 2 · n , but here we scale the distances up by n to achie ve “typical norms” to be ≈ n , which is the dimension. 5 Examples. Many n atural metric probab ility space s ar e Normal Lévy families. For example, all the following examples u nder normalized d istance (to make the typical norms ≈ n ) are normal L é vy families as stated in Defin ition 3.1: the unit n -sphere with un iform distribution u nder the Euclidea n or geodesic distance, R n under Gaussian distribution and Euclidean distance, R n under Gaussian distribution and Euclidean distance, th e unit n -cub e and u n it n -ball under the uniform distribution and Eu clidean distance, any product distribution of dimension n unde r the Hamming distance . See (Ledou x, 20 01; Giannopo u los & Milman, 200 1; M ilman & Schechtm an, 1986) for more examples. The following lemma was proved in Mahlou jifar et al. (20 18b) when Normal Lévy input spaces. Lemma 3.2. Let the inpu t spa ce of a h ypothesis classifier h b e a Normal Lévy family ( X n , d n , D n ) i ∈ N . If the risk of h with r espect to th e gr ound truth concep t function c is bigger than α , Risk ( D n , c, h ) ≥ α , and if an ad v e rsary A ca n perturb instances by u p to b in metric d n for b = p n/k 2 ·  p ln( k 1 /α ) + p ln( k 1 /β )  , then the adve rsarial risk is AdvRisk A ( D , h, c ) ≥ 1 − β . Definition 3.3 ( α -close function families) . Suppose D is a distribution over X , and let C be a set of function s fr o m X to some set Y . W e call C α -close with respect to D , if there are c 1 , c 2 ∈ C such that Pr x ← D [ c 1 ( x ) 6 = c 2 ( x )] = α . Examples. The set of homog eneous h alf spaces in R n are α -clo se for all α ∈ (0 , 1] und er any o f the fo llowing natural distributions: u niform over the u nit sph ere, u niform inside the un it ball, and isotropic Gaussian. This can be proved by picking tw o half sp aces that their disagreemen t region under the mentioned distributions is exactly α . T he set o f (mo notone, or no t necessarily mo noton e ) conjunc tio ns are α -close for α = 2 − k for all k ∈ { 2 , . . . , n } under the u niform distrib ution over { 0 , 1 } n . This c an be pr oved by lo oking at c 1 = x 1 ∧ . . . ∧ x k − 1 and c 2 = x 1 ∧ . . . ∧ x k − 1 ∧ x k = c 1 ∧ x k . Since all th e variables that appear in c 1 also ap pear in c 2 , we have th at Pr x ←{ 0 , 1 } n [ c 1 ( x ) 6 = c 2 ( x )] is eq ual to Pr x ←{ 0 , 1 } n [( c 1 ( x ) = 1) ∧ ( c 2 ( x ) = 0)] , and as a consequence this is equal to 2 − ( k − 1) − 2 − k = 2 − k . W e now state and prove our m ain results. Theorem 3 .4 is stated in the asymptotic for m considering attack families that attack the p roblem for sufficiently large index n ∈ N of the p r oblem. W e describe a quan titati ve variant af terwards (Lemma 3.5). Theorem 3.4 (Limits o f adversarially robust P A C learning ) . Sup pose P n = ( X , Y , C , D , H ) is a r ealizable classification pr oblem and that X is a No rmal Lévy F a mily (Definition 3.1) over D and a metric d , and that C is Θ( α ) -close with r espect to D for all α ∈ [2 − Θ( n ) , 1 ] . Then, the fo llowing hold even for P AC learning with parameters ε = 0 . 9 , δ = 0 . 49 . 1. Sample comple xity of P A C lear ning robust f o e vasion attacks: (a) Exponential lower bound: Any P AC learning alg orithm that is r obust aga inst all attacks with a su blinear tampering b = o ( n ) budget under the metric d requir es expo- nential sample complexity m ≥ 2 Ω( n ) . (b) Super -polynomial lower bound: P AC learning tha t is r ob ust against against all tam- pering attacks with budget b = e O ( √ n ) , r equires at lea st m ≥ n ω (1) many samples. 2. Ruling out P A C learning r ob ust to hybrid attacks: Suppo se the tampering b udget of the evasion adversary can be any b = e O ( √ n ) , and let B λ be any class of po isoning attacks that can r emove λ = λ ( n ) fraction o f th e training examples for an (arbitrary small) inverse po lynomial λ ( n ) ≥ 1 / po ly( n ) . Let R b e the class of hybrid attacks that first do a p oisoning by some B ∈ B λ and then an evasion by some adversary of b udget b = e O ( √ n ) . Then, P n is n ot P AC learnable ( re gar dless of sample complexity) u n der hybrid attacks in R . As we will see, Part 1a and Part 1b of Th eorem 3 .4 are special cases of the following m ore quantita- ti ve lower bo und that might be of inde penden t in terest. Lemma 3. 5. F or the setting of Theo rem 3.4, if the tampe rin g budget is b = ρ · n , for a fixed functio n ρ = ρ ( n ) = o (1) , then any P A C learning algorithm for P n under evasion attacks of tamp ering budget b = b ( n ) , even for p arameters ε = 0 . 9 , δ = 0 . 49 requir es sample co mplexity at least m ( n ) ≥ 2 Ω( ρ 2 · n ) . 6 Examples. Here we list some n atural scenario s that fall into the conditio ns of Th eorem 3.4. A ll examples o f Nor m al Lé vy families listed after D e fin ition 3.1 tog ether with the con c ept c la ss of half spaces satisfy the co nditions of Theore m 3. 4 an d hence cann ot b e P AC lear n ed using a p oly ( n ) num- ber o f sam ples. Th e r eason is th at one can always find two half spaces whose sym metric difference has measure exactly ε . Moreover , as discussed in examples following Defin ition 3.3, even d iscrete problem s such as le a rning mo notone - conjun ctions und er the uniform distribution (and Hamming distance as p erturbatio n metric) fall into the cond itions of Theor e m 3.4, for which a lo wer b ound on their sample comp lexity (or even impo ssibility) of robust P AC learning could be ob ta in ed. Remark 3.6 (Evasion-robust P A C learning in th e RAM computing m odel with real n umber s) . W e remark that if we allow (truly) real number s rep resent the concept and hyp othesis classes, one can ev en rule out P A C learning (not just lower bound s on sample complexity) under similar perturb a- tions describe in Part 1. I ndeed, by inspe cting the same proof of Theo rem 3 .4 for Part 1 one can get such r esults, e.g., for learning half-spaces in dimension n wh en in puts c o me f rom isotropic Gau ssian. Howe ver , we emphasize that such (seemin g ly) stron ger lower bo unds ar e not realistic, as in real set- tings, we eventually work with fin ite precision to r epresent the c o ncept functions (of half spaces). This ma kes the set of conce p t functions finite , in which case the test error eventually reaches zer o , using perh aps exponentially many samples. Th e orem 3.4, howev er, has the useful feature tha t it ap- plies even in those settings, as long as the co ncept f unctions are rich e n ough to allow the su fficiently close (but n ot too close) pairs un der the distribution D accor ding to De finition 3.3. In what follows, we will first pr ove Lem ma 3.5. W e will then use Lem ma 3.5 to prove T h eorem 3.4. Pr oof of Lemma 3.5. Let m = m (0 . 9 , 0 . 49 , n ) be the sam p le c o mplexity of the (pr e sumed) learn er L th at achiev es ( ε, δ ) -P A C lear ning fo r ε = 0 . 9 , δ = 0 . 4 9 . If m = 2 Ω( n ) already , we are do ne, as it is ev en larger than what Lemma 3. 5 states, so let m = 2 o ( n ) , and we will derive a contradictio n . Since the distribution D is fixed, in the d iscussion below , we simply deno te Risk ( D , h, c ) as Risk ( h, c ) . Recall th at, by assump tion, for all ε ∈ [2 − Θ( n ) , 1 ] , there are c 1 , c 2 ∈ C that are Θ( ε ) -clo se under the distribution D . Because m = 2 o ( n ) , it ho lds that 1 /m ≥ ω (2 − Θ( n ) ) , and so there are c 1 , c 2 ∈ C such that for ∆( c 1 , c 2 ) = { x ∈ X | c 1 ( x ) 6 = c 2 ( x ) } we have Ω  1 m  ≤ Pr x ← D [ x ∈ ∆( c 1 , c 2 )] ≤ 1 100 m . Now , consider m i.i.d . samp les that a r e g iv en to the learn e r L as a training set S . W ith probab ility at least 0 . 99 of the samp lin g o f S , all x ∈ S would b e ou tside ∆( c 1 , c 2 ) , in which case L would have no way to distinguish c 1 from c 2 . So, if we pick c ← { c 1 , c 2 } at random and pick test instance x ← ( D | ∆( c 1 , c 2 )) , the hypothesis h = L ( S ) fails with probability at least 0 . 99 / 2 . Thus, we can fix the cho ice o f c ∈ { c 1 , c 2 } , such that with p robab ility 0 . 99 / 2 > 0 . 49 we get a h ← L ( S ) where Risk ( h, c ) = Pr x ← D [ h ( x ) 6 = c ( x )] ≥ 1 2 · Pr x ← D [ x ∈ ∆( c 1 , c 2 )] ≥ Ω  1 m  . For this fixed c an d any such learned hypothesis h with Risk ( h, c ) = Ω(1) /m , by Lemm a 3. 2, the adversarial risk reaches AdvRisk A b ( h, c ) ≥ 0 . 99 by an attack A ∈ A b that has tampe ring budget: b = O ( √ n ) ·  p ln( O ( m )) + p O (1)  ≤ t · ( √ n · ln m ) for u niversal constant t . But, we said at the beginnin g that the tampering budget of the adversary is ρ ( n ) · n . Ther efore, it sho uld be that ρ ( n ) · n < t · ( √ n · ln m ) , as otherwise the evasion-robust P AC learner is not actua lly robust as stated. Thus, we get m ≥ e ρ ( n ) 2 · n/t = 2 Ω( ρ ( n ) 2 · n ) which finishes the pro of of Lemma 3.5. W e now prove Theo rem 3.4 using Lemma 3.5. Pr oof of Theor em 3.4. Using Lemma 3.5, we will first prove Part 1a, then Part 1b, and then Part 2. Throu g hout, ε = 0 . 9 , δ = 0 . 49 are fixed, so the sam ple complexity m = m ( n ) is a function of n . 7 Proving Part 1a. W e claim that P AC learnin g resisting all b = o ( n ) -tamp ering attacks r equires sample complexity m ≥ 2 Ω( n ) . The reason is that, otherwise, there will be an infin ite sequence of values n 1 < n 2 < . . . for n for which m = m ( n i ) ≤ 2 γ ( n i ) · ( n i ) for γ ( n ) = o (1) . Howe ver , in that case, if we let ρ ( n ) = γ ( n ) 1 / 3 , because ρ ( n ) = o ( n ) , by Lemma 3.5, the sample com plexity is m ( n i ) ≥ 2 Ω( ρ ( n i ) 2 · n i ) = ω  2 γ ( n i ) · n i  . Howe ver , this is a c ontradictio n as we pr eviously assumed m ( n i ) ≤ 2 γ ( n i ) · ( n i ) . Proving Part 1b. Supp o se the a d versary can tampe r instances with b udg et b ( n ) = κ ( n ) · √ n f or κ ( n ) ∈ polylo g ( n ) . Since we can rewrite b ( n ) = ρ ( n ) · n for ρ ( n ) = κ ( n ) / √ n , then by Lemm a 3.5, the sample comp lexity of L shou ld be at least m ( n ) ≥ 2 Ω( ρ ( n ) 2 · n ) = 2 Ω( κ ( n ) 2 ) . Therefo re, if we cho o se κ ( n ) = log( n ) 2 , the sam ple complexity of L b ecomes m ≥ n log n ≥ n ω (1) . Proving Part 2. Let be c 1 , c 2 ∈ C be such that for ∆( c 1 , c 2 ) = { x ∈ X | c 1 ( x ) 6 = c 2 ( x ) } we have Ω( λ ) ≤ Pr x ← D ( c 1 ,c 2 ) [ x ∈ ∆( c 1 , c 2 )] ≤ λ. Consider a poison ing attacker A 1 that given a d ata set S , it r emoves any ( x, y ) fr o m S such that x ∈ ∆( c 1 , c 2 ) . No te that the (expected) num ber of such examples is Pr[ x ∈ ∆( c 1 , c 2 )] ≤ λ . Let e S be the modified tr aining set. Th e lear ner L ( e S ) now h as now way to distinguish between c 1 and c 2 . Thus, like in Lemm a 3 .5, we can fix c ∈ { c 1 , c 2 } , such that L ( e S ) always pro duces h where Risk ( h, c ) = Pr x ← D [ h ( x ) 6 = c ( x )] ≥ 1 2 · Pr x ← D [ x ∈ ∆( c 1 , c 2 )] ≥ Ω( λ ) . For this fixed c an d any such learned hypothesis h with Risk ( h, c ) = Ω( λ ) , by Lemm a 3.2, the adversarial risk (under attacks) r eaches AdvRisk A b ( h, c ) ≥ 0 . 99 b y an attack A ∈ A b that changes test instances x by at most b f o r b = O ( √ n ) ·  p ln( O (1 /λ )) + p O (1)  ≤ O ( p n · ln(1 /λ )) . Since λ = 1 / po ly( n ) , it h o lds that b = e O ( √ n ) . 4 Extensions In this section, we d escribe some extensions to Theorem 3.4 in various dir ections. Extension to randomized predictors. In Th eorem 3.4, we ruled ou t P A C learning (or its small sample complexity) even for very large values ε = 0 . 9 , δ = 0 . 49 . One migh t argue that proving such lower boun d could not b e impossible be c ause a trivial h ypothe sis (f or the setting where Y = { 0 , 1 } ) can achieve ε = 0 . 5 by outp u tting ran dom bits. However , th is trivial pr edictor is randomized , while Theorem 3.4 is proved for deterministic hypotheses. For the case of randomized hy potheses, one can adjust the pro of of Theorem 3.4 to get similar lo wer b ound s for ε = 0 . 49 , δ = 0 . 49 a s follows. In the pro of o f T heorem 3 .4 we first showed that small sample com p lexity implies th e existence of c that with probability > 0 . 49 it will have an error region with a non -negligible measure. When the hypothe sis is rand omized, howe ver , we cann ot w ork with the traditiona l no tion of error region, because on e very p oint x ∈ X , th e hypothesis could be wrong h ( x ) 6 = c ( x ) with some pro bability in [0 , 1 ] . W e can, howe ver , work with the relaxed notio n of “app roximate error” region, d efined as AE ( h, c ) = { x | Pr h [ h ( x ) 6 = c ( x )] ≥ 1 / 2 } , where th e probab ility is ov er the randomn e ss of h . In pr oofs of both Lemma 3.5 and Theorem 3 .4 we deal with two close concept fu nctions c 1 , c 2 that are “indistinguishab le ” for the hypothesis h and then conc lude that fo r each point x ∈ ∆( c 1 , c 2 ) , h makes a mistake on at least one of c 1 , c 2 . If h is ran domized , we canno t say this a nymore, b ut we can still say that for each such po int x ∈ ∆( c 1 , c 2 ) , for a t least one of c 1 , c 2 , h ( x ) is wrong with prob ability at least 0 . 5 . Therefo re, we get the same lower bound on the size of the AE as we got in Lemma 3.5 an d Theorem 3 .4. Howe ver , expanding the set AE in stead of an actual error- region, imp lies that the ad versarially perturb ed po ints e x that fall into AE are now misclassified with probab ility 0 . 5 . Th us, at least 0 . 99 fraction of inputs can be p erturbed into AE to be misclassified with probab ility > 0 . 49 . 8 Lower bound for P A C learning o f a “typical” concept function. Theorem 3.4 only proves the exis tence o f at least o ne concept func tion c ∈ C for which the (presumed) robust P A C learner will either fail (to P A C lear n ) or will need large samp le complexity . Now , su p pose conc e pt functions themselves come from a (n a tural) distribution and we o nly want to ro bustly P A C learn mo st of them . Indeed , we can e xtend the p r oof of Theorem 3.4 to show that for natural settings, the impossibility result extends to at least half of the concept fun ctions, not just a few p athologic a l cases. T o extend Th eorem 3.4 to the more general “typical” failure over c ← C (stated as Claim 4.2 below) we need the following definition as an extension to Definition 3.3. Definition 4 .1 (Un iformly α -close function families) . Suppose D is a distribution over X , and let C be a set of fu nctions from X to some set Y . W e call C uniformly α -close with respect to D , if there is a joint distribution ( c 1 , c 2 ) where b oth coordinates a r e uniformly distrib uted ov er C , and that f or all ( c 1 , c 2 ) ← ( c 1 , c 2 ) , it bo th ho ld s that c 1 , c 2 ∈ C and that Pr x ← D [ c 1 ( x ) 6 = c 2 ( x )] = α . Claim 4.2. In Theor em 3.4 and Lemma 3.5, make the only change in the setting as follows. The concept class C now satisfies the str onger condition o f b eing u niform α - c lose with respect to D . Then, the same limitation s of P AC learning hold for at lea st measur e half of c ← C . Here we sketch why Claim 4.2 holds. The difference is that now , instead of knowing the existence of an α -close p air ( c 1 , c 2 ) , we h ave distrib ution ( c 1 , c 2 ) samp les from wh ich satisfy the α -clo se proper ty . Therefor e, for all samples ( c 1 , c 2 ) ← ( c 1 , c 2 ) , at least one of c 1 or c 2 is “bad” for the (presumed ) P AC learn er L (with the same p roof befo re). But, since ea c h of th e coordin ates in ( c 1 , c 2 ) is marginally uniform , therefor e, at least m easure 1 / 2 of c ← C is b a d f or L . Example. Consider the uniform measure over homog eneous half spaces in dimension n a s the set of c oncept fun ctions C : choo se a point w in the unit sphere and select the half space { x | h x, w i ≥ 0 } . It is easy to see that C with such me a sure is uniformly α -close with respect to the isotrop ic Gaussian distribution (o r unifor m distribution over the unit sph e re). Thu s, Claim 4.2 ap plies to this case. 5 Conclusion and Open Questions W e examined ev asion attacks, where the adversary c a n per turb in stan ces during test time, as well as hybrid attack s where the adversary can p erturb instance s du r ing bo th trainin g and test time. F or ev a- sion attacks we gave an expo n ential lower boun d o n the sample complexity even wh en the adversar y can pertu rb instances by an amou nt of o ( n ) , wh e re n is the data dimension capturing the “typical” norm of an input. For hybrid attacks, P AC learning is ruled out altogether when the adversary can poison a small fraction of the training examples and still perturb the test instance by a sublinear amount o ( n ) (or even e O ( √ n ) ). Our result shows a d ifferent behavior wh en it comes to P A C lear ning fo r error-region adversarial risk compare d to previously used no tions of adversarial robustness based on corrup ted in puts. In particu - lar , in the error-region variant o f adversarial risk, r e alizable prob lems stay re a liza ble, as normal risk zero f o r a hypothe sis h also implies (error-region) adversarial risk zero f or the same h . Th is makes our results mor e striking , as they app ly to a g nostic learning as well. Open questions. Our The orem 3.4 relies on a level of tamper ing to be at least e O ( √ n ) to imply the super-polynom ial lower bou nds. One n atural question is to find th e exact threshold of pertur bations needed that trigger s super-polynomial lower b ounds on sample complexity . Another important directio n is to stud y th e sample complexity of P AC learning ( with concrete pa- rameters ε, δ ) for p ractical distributions such as images or voice. O u r lo wer bou nds of th is work are only proved for th eoretically natu ral distributions that ar e provably concentrated in h igh d imension. Mahloujifar et al. (2019), presents a m ethod f o r em pirically ap proxim ating th e concentratio n of such distributions given i.i.d . samp les fr o m them. Finally , we ask if similar results co uld b e proved for corrupted- input ad versarial risk. Note that previous work study in g learnin g u nder co rrupted -input adversarial risk (Bubeck et al., 2018b; Cullina et al., 2018; Feige et al., 2018; Attias et al., 2 018; Khim & Loh, 201 8; Y in et al., 2018; Montasser et al., 201 9) focus on ag nostic learning, b y aiming to get close to the “b est” ro bust classi- fier . Howe ver , it is no t clear how g o od the best classifier is. It remains open to find o ut when we can learn robust classifiers (under corrupted -input risk) in wh ich th e total adversarial risk is small. 9 Refer ences Attias, I. , Kontorovich, A., and Mansou r , Y . Improved gener alization bounds for robust learn ing. arXiv pr eprint arXiv:18 10.02 180 , 2018. Balcan, M.-F . and L ong, P . Activ e an d passive lea r ning of linear separators u n der log -concave distributions. In Conference on Lea rning Theory , pp. 28 8–316 , 2013. Biggio, B., Nelson, B., an d Laskov , P . Poisonin g attacks against support vector machines. In Pr o - ceedings of the 29th International Cofer ence on I nternationa l Con fer ence on Machine Lea rn ing , pp. 146 7–147 4. Omnipress, 2012. Biggio, B., Corona, I . , Maior c a, D., Nelson , B., Sr n dic, N. , Laskov , P ., Giacinto, G., and Roli, F . Evasion Attacks a gainst Machine Learning at T est T ime. In ECML/PKDD , pp . 387–4 02, 201 3. Biggio, B., Fumera, G., a n d Roli, F . Security evaluation of patter n classifiers under attack. IEEE transactions o n knowledge and data eng ineering , 2 6(4):9 84–9 9 6, 20 14. Bshouty , N. H., Eir o n, N., an d Kushilevitz, E. P A C learn in g with nasty noise. Theoretical Compu ter Science , 288( 2):255 – 275, 2002. Bubeck, S., L ee, Y . T ., Price, E., and Razen shteyn, I. Adversarial examples fr om cr y ptogra phic pseudo- r andom gener ators. a rXiv pr eprint arXiv:181 1.064 18 , 20 18a. Bubeck, S., Price, E., and Razenshteyn, I. Adversarial examples fro m computation al con straints. arXiv pr eprint arXiv:18 05.10 204 , 2018b. Carlini, N. and W agner, D. A. T owards Evaluating the Rob ustness of Neural Networks. I n 2017 IEEE Symposium on Security a nd Privacy , S P 201 7, San Jose, CA, USA, May 2 2-26, 2017 , pp. 39–57 , 201 7. Cullina, D., Bhagoji, A. N., and Mittal, P . Pac-learning in the p resence of evasion ad versaries. arXiv pr eprint arXiv:18 06.01 471 , 2 018. Degwekar , A. and V aik untanatha n , V . Compu tatio nal limitatio n s in robust classification an d win-win results. arXiv pr eprint arXiv:190 2.010 86 , 2 0 19. Diochnos, D., Mahloujifar , S., an d Mahmoo dy , M. Adversarial risk and robustness: General defini- tions and implications for th e uniform distribution. In Advan c es in Neural Information Pr ocessing Systems , pp. 103 59–1 0 368, 2018. Feige, U., Man sour, Y ., an d Schap ire, R. L earning an d inferenc e in the p resence o f corrup ted inputs. In Confer ence on Learning Theo ry , p p. 637– 657, 20 15. Feige, U., Mansour, Y ., and Schapir e, R. E. Robust inf e r ence for multiclass c lassification. In Algorithmic Learning Theo ry , pp. 3 68–38 6, 2018 . Ford, N., Gilmer , J., Carlini, N., and Cubuk, D. Adversarial examp les are a natural con sequence of test error in n oise. arXiv pr eprint arXiv: 1 901.1 0513 , 2019 . Giannopo ulos, A. A. an d Milman, V . D. Eu clidean struc tu re in finite dim ensional no rmed spaces. Handbo ok of the geometry of Ba nach spaces , 1 :7 07–7 79, 200 1. Gilmer , J., Metz, L., Faghr i, F ., Schoenholz, S. S., Raghu, M., W atten berg, M., an d Goodfellow , I. Adversarial spheres. arXiv pr eprint arXiv:1801 .0277 4 , 2018. Goodfellow , I., Shle n s, J., and Szegedy , C. Exp laining and Harnessing Adversarial Exam ples. In ICLR , 2015. URL http:// arxiv.org/ab s/1412.6572 . Gu, T ., Dolan-Gavitt, B., and Garg, S. Badnets: Identif ying vulnerab ilities in the machine learning model supply cha in . arXiv preprint a rXiv:1708 .0673 3 , 201 7. Kearns, M. J. an d L i, M. L earning in the Presence of M alicious Errors. SIAM Journal on Computing , 22(4) :807–8 37, 19 93. 10 Khim, J. and Lo h, P .-L. Adversarial risk bound s for binar y classification via function transfo rmation. arXiv pr eprint arXiv:18 10.09 519 , 2018. Ledoux , M. The Concentration of Measure Pheno menon . Number 89 in Mathematical Surveys and Monog r aphs. American Mathem atical Society , 2001 . Long, P . M. On the samp le complexity of p ac learnin g half-space s against th e uniform distribution. IEEE T ransaction s on Neural Networks , 6(6):1556 –155 9 , 1995. Madry , A., Makelov , A . , Schmid t, L., Tsipras, D., and Vladu , A. T owards deep learning models resistant to adversarial attacks. arXiv pr eprint arXiv:1 7 06.06 083 , 2017. Mahloujifar, S., Dio chnos, D. I., and Mah moody , M. Learning under p -T am p ering Attacks. In ALT , pp. 572 –596, 2 018a. Mahloujifar, S., Diochnos, D. I., and M ahmood y , M. The curse of co ncentratio n in robust learn ing: Evasion a n d poisonin g attacks from conce n tration of measure. arXiv preprint a rXiv:1809 .0306 3 , 2018b . Mahloujifar, S., Zhan g, X., Mahm o ody , M., and Ev ans, D. Empir ically me asuring concentration : Fundamen tal limits on in trinsic robustness. Safe Machine Learning workshop at ICLR , 2019. Milman, V . D. a n d Schechtma n , G. Asympto tic th e ory of finite dimension al n ormed spaces , volume 1200. Sp ringer V erlag, 198 6 . Montasser, O., Han neke, S., and Srebr o, N. Vc classes are ad versarially robustly learn able, but only improp e rly . arXiv preprint arXiv : 1902. 04217 , 201 9. Papernot, N., McDa niel, P ., Sinha, A., and W ellman , M. T owards the science of security an d priv acy in machine learning . arXiv preprint a rXiv:1611 .0381 4 , 201 6a. Papernot, N., McDaniel, P . D., W u, X., Jha, S., an d Swami, A. Distillation as a Defense to Adversar- ial Pertur bations Against Deep Neural Networks. In IEEE Sym p osium on Security and Priva c y , SP 2016 , San Jose, CA, USA, May 22- 26, 2 016 , pp. 582– 597, 20 16b. Sabato, S., Srebro, N., and T ishby , N. Distribution-d epende n t sample complexity of large margin learning. The Journal o f Machine Learning R esear ch , 14 (1):21 19–21 49, 201 3 . Schmidt, L., Santurk ar , S., Tsipras, D., T alwar , K., and Madry , A. Ad versarially Robust Generaliza- tion Requires More Data. arXiv preprint a rXiv:1804 .1128 5 , 20 18. Shafahi, A., Hua n g, W . R., Najibi, M., Suciu , O. , Studer, C., Du mitras, T . , and Goldstein, T . Poison f rogs! targeted clean-lab el po isoning attacks o n ne u ral networks. a rXiv preprint arXiv:180 4.007 92 , 20 18. Szegedy , C., Zaremba, W ., Sutskever , I., Brun a, J., Erhan , D., Good fellow , I., and Fergus, R. Intriguin g pr operties of neural networks. In ICLR , 201 4 . URL http://arxiv .org/abs/131 2.6199 . Tsipras, D., Santurkar, S., En g strom, L ., Turner , A., and Madry , A. Robustness may be at odd s with accuracy . stat , 1 050:11 , 201 8. V aliant, L. G. A Theo ry of the Learnab le. Communicatio ns o f the AC M , 27 (11):1 1 34–1 142, 1984. V aliant, L. G. Lear n ing d isjunctions of conjun ctions. In IJCAI , p p. 560–56 6, 1985 . Xu, W ., E vans, D., an d Qi, Y . Feature Sq ueezing: Detectin g Adversarial Ex amples in Deep Neural Networks. CoRR , ab s/1704. 0 1155 , 20 17. Y in , D., Ramchand ran, K., and Bartlett, P . Radem acher co mplexity fo r a dversarially ro bust general- ization. arXiv pr eprint arXiv:1 810.1 1914 , 2018. 11