Identifying the Most Explainable Classifier

Iden tifying the Most Explainable Classifier Brett Mullins brettcm ullins@gmail.com Octob er 24, 2019 Abstract W e in tro duce the notion of p oin t wise co v erage to measure the explainabilit y prop- erties of machine learning classifiers. An explanation f o r a pr ed i ction is a defin ab ly simple region of the feature space sharing the same lab el as the p redicti on, and the co v erage of an explanation measures its size or generalizabilit y . With this notion of explanation, we inv estiga te whether or not there is a natural c haracteriza tion of the most explainable classifier. Ac cording with our in tuitions, we pro v e that the binary linear classifier is un iquely the most explainable classifier up to negligible sets. 1 In tro duct i o n The interpretabilit y of machine learning mo dels and explanations of mo del predictions hav e receiv ed m uc h atten tion o ve r the past decade [4]. These approache s attempt to explain what influences a mo del’s b eha vior on a particular observ ation [10]. Though these concepts are often equiv o c ated, in terpretabilit y usually fo cuses on what one can learn b y insp ec ting a mo del’s structure, e.g., observing the sign and magnitude of a w eigh t in a linear regression [8]. In con tra st, an explanation is a reason for a mo del’s b eha vior at a specific p oin t in the feature space and is lo cal to that observ ation [2]. When w e talk of a mo del b eing in terpretable, we mean that its b eha vior is transparent with resp ect to insp ecting a mo del’s structure, hence the common white-b o x/black - box dic hotom y . When w e sp eak of a mo del b eing explainable, w e mean that a reason can be giv en for the model’s b eh avior at a giv en p oin t in t he feature space t ha t is of a sufficien t generality for the contex t of mo del usage. Explainabilit y is a desirable prop ert y for a mo del, sinc e it allo ws the user to build trust that the mo del’s predictions accord with background kno wledge, to b etter understand the mo del’s b eha vior, and to ensure algorithmic fairness when used for p oten tially conseq uential decisions [1 8]. In this pap er, w e explore the notion of the mos t explainable classifier throug h represen ting classifiers as partitions of euclidean space. In contrast to algorithmic approaches to measuring explainabilit y , we in tro duce a theoretical framew ork where explainabilit y is expressed as a geometric and to pological prop ert y of a partition of euclidean space. In par t icular, w e adopt the notion of p oin t wise co v erage, first in tro duced with the pro babilis tic anc hors approach in [11], as an aggregate measure of explainability ov er all p oin ts in the feat ure space. Using 1 the notion o f p oin twis e cov erage, w e prov e a characterization result unique ly identifying the most explainable classifier as a refinemen t of the binary linear classifier. Though this result is unsurprising, it prov ides a foundatio n for our intuitions ab out linear classifiers and corrob orates the utility of this theoretical f r ame work. This pap er pro ceeds as follows. In Section 2, w e dev elop a forma l framew ork to represen t classifiers, and, in Section 3, we in tro duce an approac h to explanations of classifier predictions and a measure of classifier explainabilit y called p oin t wise cov erage. In Section 4 , w e in tro duce the refined linear classifier a nd pro ve that no classifier is more exp laina b le than it with r esp ect to p oint wise cov erage. In Section 5, we prov e the con v erse result and establish that the refined linear classifier is uniquely t h e most explainable classifier. In Section 6, w e c haracterize the collection o f classifiers that can b e refined to the refined linear classifier. F inally , in Section 7, w e conclude. 2 A F ormal Approac h to Classi fiers In mac hine learning and related fields, the general task of classification is to accurately assign an observ ation to its corresp onding lab el. In this section, we presen t a for ma l framework to express classifiers as partitions of euclidean space. W e define a c lassifier P as a partition of R n suc h that there exists R ∈ P , called the r efinement set , where R is p oten tially empt y , meagre, and Leb es gue n ull. D e fine the lab el set of P as L P = P \ { R } and the fe at ur e s p ac e of P as S L P . F or x ∈ R n , let P ( x ) ∈ P b e the mem b er of P con taining x . W e call P ( x ) the la b el of x with resp ec t to mo del P . W e call a classifier trivial if the lab el set is a singleton set; o t herwise, the classifier is non-trivial . W e sp ec ify that the refinemen t set R ⊂ R n is b oth meagre and Leb esgue n ull to capture that R is small or negligible b oth top ologically a nd pro babilis tically . This sp ecifi cation follo ws from the in tuitio n in the case where R is the b oundary b et we en tw o lab els of a classifier, e.g., if R is the h yp erplane separating the t wo lab els of a binary linear classifie r. Recall that a set is me ag r e if it can b e represen ted as the coun table union of no where dense sets and a set B is nowher e dense if the closure of B , B , has no no n-trivial op en set. On the other hand, for probability measure µ , w e hav e that µ ( R ) = 0, since µ ( R ) = R R f dλ and λ ( R ) = 0 where λ is the Leb esgue measure a n d f is a densit y function o ver R n . Observ e tha t w e mak e the simplifying assumption that the data generation pro ce ss is con tin uous; while this is not strictly general, it is reasonable given that discrete features are em b edded in R n and treated as n umerically con tin uous in man y p opular mac hine learning mo dels and algorit hms. As an example, let us consider the classifier P in Figure 1. F or this classifier, L P = { C , D , E , F } . F rom this figure, we can consider m ultiple classifiers. One suc h classifier is giv en by P = L P ∪ {∅} where E , F contain their b oundaries but C , D do not. In this case, the refinemen t set is empt y . A classifier with a n empty refinemen t set is called or dinary ; otherwise, a classifier is a r efinemen t of some ordinary classifier and is called r efine d . An example of a refined classifier from the figure ab o v e is giv en b y Q = { C , D , E , F , R } where L Q = { C , D , E , F } , no member of L Q con tains their b oundary , and R = S L ∈ L Q bd ( L ). 1 While w e only ev er see ordinary classifie rs “in the wild”, the introduction of the notion of refine- men ts r e mov es the artificial complexity generated by edge cases in the feature space. By 2 C D E F x 1 x 2 Figure 1: Example Classifier as a P artition of R 2 mo ving to a refinemen t of a classifier, we can b etter assess the ag gregate explainabilit y and top ological prop erties of that classifier when edge cases are pr esen t. The framew ork dev elop ed in this section is sufficien tly general to represen t a n y classifier with con tinuous features. This ranges from a binary linear classifier where the lab e ls are op en and closed halfspaces in the feature space, resp ectiv ely , to decision trees where t he lab els ar e disjoin t unions of conv ex p olytopes, i.e., in tersections of op en a n d closed halfs- paces. These mo dels are w ell-studied and hav e simple geometric characterizations. As w e increase complexit y with, for example, neural net w or ks, w e find that represen tations of these classifiers within this framework are p ossible but not in tuitive or clear due t o the comp o- sitions o f non-linear activ ation functions found in many neural net w ork a rc hitectures [13]. Nonetheless, in the next section, w e intro duce the p oin twise co v erage approach to measuring the explainabilit y o f a classifier expressed as a partition of euclidean space in the framew or k dev elop ed thus fa r. 3 Explainabilit y b y P oint wise Co v erage An explanatio n of a classifier prediction is an elusiv e concept. Ideally , an explanation pro- vides a reason for why a classifier assigns a particular lab el to a giv en observ ation. One w ay to ac hiev e this and the p ersp e ctive w e adopt in this pap er iden tifies an explanation for a classifier at a give n o bserv a t io n as a definably simple region of the f e ature space con taining the observ atio n where a ll p oin ts in the region are assigned the same lab el. W e refer to these definably simple regions o f the feature space as anchors. T o what exten t do es defining suc h a region of the feature space pro vide an explanation for the mo del’s classification? A t first pass, we can think of an anc hor as a suffi cien t condition for the classific at io n of a p oin t in the feature space; ho w ev er, that alone is unhelpful, since an anc hor could be an a r b itra ry subset of a lab el. By adding the requirem ent that anchors b e definably simple regions of the featur e space, we can ensure that the p oin ts in the anchor are meaningfully related or related b y a sim ple condition. An anc hor for a given p oin t acts as an explanation by providing the definition of the relation grouping t h e p oin ts in the anchor as the reason for the classification. 1 Note that bd ( X ) = X ∩ R n \ X is the b oundary of the set X ⊆ R n . Whenever p ossible we follow the notation conven tions in [9]. 3 Just as there are many approache s to interpretabilit y and explainabilit y , there are man y w a ys to sp ecify what is meant f or an anc hor to b e definably simple. As an example, the probabilistic anc hors approac h uses rectangles in the f e ature space that minimize the n umber of conditions sp ec ified as anc hors [11]. In con trast, w e adopt op en balls in euclidean space as anc hors. Observ e tha t b oth of these a ppro ac hes use a distance-based relation to gro up po in ts in the resp ectiv e anc hors, so tha t p oin ts in an anc hor are in some sense spatially close to one another. While the probabilistic anc hors a ppro ac h is largely concerned with algorit hmic and computational pro perties of iden tifying anc hors in the feature space [14, 3], w e fo cus on geometric and to pological prop erties of the lab el set. In particular, for a classifier P , w e define an anchor for a p oin t x ∈ R n as a n op en ball A = B ( c, r ) such that x ∈ A ⊂ P ( x ). Notice that the anchor need not b e cen tered at the observ ation of interes t. Observ e that op en balls are basic op en sets in the standard top ology o n R n . F rom a definabilit y p ersp ec tive, basic op en sets are among the most simple sets of a top ology , since all other o pen sets are coun table unions of basic op en sets. Moreo v er, the op en sets o ccup y the space at t he b ottom of t he Borel hierarch y , a stratification of the Borel sets, i.e., the sets constructed f r o m open sets by iterative application of coun table union, countable in tersection, and complemen tatio n , and ordered b y their definability in terms of o pen sets. T o denote that the o pen sets are definably simple, w e say t ha t the op en sets hav e a B or el r ank of 1. Sets of greater Bo r el rank are then more definably complex. 2 Explanations are usually lo cal in the sense that they do not a pp ly to all p oin ts in the feature space. F or example, an explanation for the classification o f a p oin t x ∈ R n b y classifier P need not b e an explanation for the class ification of a distinct p oin t y ∈ R n . In particular, this will b e the case when y do es not b elong to a n anc hor for x irresp ectiv e of whether P ( y ) = P ( x ) or not. W e may mak e the notio n of lo cal explanations precise by in tro ducing the co v erage of an anc hor. F or an anc hor A for p oin t x ∈ R n with radius r A with resp ec t to classifier P , the c over ag e of A is giv en b y c P ( A ) = r A > 0. If there exists an anchor A = B ( c, r ) f o r a p oin t x ∈ R n with radius r > 0 then that a nc hor acts as an explanation for all p oin ts in the feature space within an r − neigh b orho o d of c . C D E F • x x 1 x 2 Figure 2: Example C l assifier with Anch ors If a p oin t in the feature space has an anc ho r , then it has many suc h anc hors. T o see this, observ e tha t if A is an anch or for x ∈ R n then there exists an r > 0 suc h that 2 Sets of greater Bor e l rank are outside of the scop e o f the present pap er. F or more on the Borel sets and the Bor el Hierar c h y , see [6, 15]. 4 B ( x, r ) ⊂ A . Clearly , B ( x, r ) is an anc hor for x ; how ev er, c P ( B ( x, r )) = r ≤ c P ( A ), since B ( x, r ) ⊂ A . Giv en that a p oin t can ha v e many anc hors and eac h are equally definably simple from a top ological p erspectiv e, w e c ho ose a n anc hor with the great est cov erage as the b es t explanation for the classification. Let A x denote the set of anc hors for x with res p ect to classifie r P . W e say that the c over age of P at x is g iven by C P ( x ) = sup A ∈A x c P ( A ). If no anchors exist for a p oin t x with resp ec t to classifier P , w e sa y that the co ve rag e of P at x is zero. If there exists a sequence o f a nchors for x with increasing un b ounded cov erage, then we sa y that t h e co v erage of P at x is infinite. Otherwise, w e sa y that the cov erage of P at x is finite. Wh y do we prefer anc hors with greater cov erage to those with less as explanations? Just as cov erage is a measure of the size of a n anc hor as a ball in euclidean space, it is also a measure o f the generalit y of an explanation in feature space. In turn, w e may say that co v erage is a measure of the strength of a reason for a classifier’s b eha vior at a p oin t in the feature space. T o illustrate this p oin t, let us consider a concern raised ab out t h e veracit y of explanations of class ifier predictions when a classifie r learns spuriously or erroneously fro m its training data [7 ]. M N • x • y x 1 x 2 Figure 3: Example of an Ov erfit Decision T ree F or the f ormer case , consider a decision tree classifier that is p oten tially ov erfit during training. The example in Figure 3 is a binary classifie r P = { M , N } , and let us assume that it is ov erfit with resp e ct to the region containing the p oin t x . Notice t ha t since eac h lab el is the disjoint union of conv ex polytop es, t his classifier represen ts a decision tree. Let us consider the cov erage of P at p oin ts x, y in the feature space . While b oth p oin ts belong to the lab el N , it is apparent that C P ( x ) < C P ( y ). The spurious learning o f the classifier is, thus, reflected in this disparity in co ve rag e b et w een the tw o p oin ts. Relat ive to y , the explanation for the classification for x is m uc h w eak er. T o address the latter case, there need not be a correlation b et w een a classifier’s explainabilit y a nd its ve racity with respect to the t r a ining data. This is to sa y that the explainabilit y prop erties o f a classifier do not necessarily imply any thing ab out it correctly learning from t he training dat a. T o compare the explainability of a classifier at t wo p o in ts in the f e ature space, w e can compare t he classifier’s co v erage at those p oin ts. Note that this comparison is alw ay s relative to the scale of t he feature dimensions; if the scale of the features are transformed, e.g., b y an affine tr ans for m atio n, then the resulting cov erage v alues ma y b e differen t, since what w ere previously anc hors may now be op en ellipsoids rather than op e n balls. Fixing the scale of 5 features, on the other hand, p ermits comparisons of cov erage and, resultan tly , explainabilit y for particular p oin ts in the feature space across v ar io us classifiers. In tuitiv ely , when one estimates the explainabilit y of a classifier or compar es the explain- abilit y of m ultiple classifiers, it is not with reference to a sp ecific p oin t in the feature space. Comparing the co v erages of v arious classifiers at ev ery p oin t in the feature space is not feasible, since tha t w ould entail uncoun tably many comparisons with no clear metho d of aggregating or summarizing the results of the comparison. A simple metho d of a g gregat- ing co v erage up to the classifier-lev el is to consid er the infim um and suprem um of cov erage across a ll p oin ts in the feature space. W e refer to these agg r ega tions of cov erage as p ointwise c over age . Let us consider tw o limiting cases of p oin twis e cov erage: zero p oin t wise cov erage and infinite p oin tw ise cov erage. W e sa y that a classifier has z e r o p oi nt wis e c over a ge if the supre- m um across all p oin ts in the feature space of a classifier’s co v erage is zero. With resp ect to co ve rag e, classifiers with zero p oin twis e co v erage are the least explainable; no p oin t in the f e ature space has an anc hor. Let us pro vide an example of suc h a classifier. Consider a classifier on R giv en b y P = { Q , R \ Q , ∅} . Since b oth la bels are dense in R , an y p oten tial anc hor fo r a p oin t in Q must contain a p oin t in R \ Q , and vice-v ersa. Luck ily , o ne is almost surely not to encoun ter such an unexplainable classifier “in the wild.” Whereas zero p oin tw ise cov erage represen t s a lo w er-b ound on explainability , infinite p oin twise cov erage is an upp er-b ound. W e sa y that a classifier has in finite p ointwise c over a g e if the infim um a c ross all p oin ts in the feature space of a classifier’s co v erage is infinite. In Section 4, w e prov e tha t there is a natural collection of classifiers that hav e this prop ert y: a refinemen t of binary linear classifiers whic h w e call r efined linear classifiers. Moreov er, in Section 5, w e pro v e that only the collection of refined linear classifiers ha ve infinite p oin tw ise co v erage. The primary result of this pap e r is a full c haracterization of infinite p oin tw ise co v erage as a classifier of the form o f a refined linear classifier: Theorem 3.1. A n on-trivial classifier P has i n finite p ointwise c over age just in c ase P is a r efine d line ar classifier. 4 Refined Linear Classifie r Linear classifiers are ubiquitous throughout the history of mac hine lear ning and in data science to da y . “The family of linear [classifiers] is one of the most useful families of h yp othesis classes, and man y learning alg orithms that are b eing widely use d in practice rely on linear [classifiers]” [1 3]. This collection of classifiers includes not j ust linear regression but lo gis tic regression, p erceptrons [12 ], linear supp ort v ector mac hines [17], etc. 3 Within the scop e of this pa p er, we are intereste d in binary linear classifiers, i.e. a linear classifier with only t wo lab els. Despite the algorithm or metho d used to tra in the classifier, a binary linear classifier can alw ay s b e r e presen ted as the w eighted sum of input features with real-v alued w eigh ts and a real-v alued threshold. With respect to po in tw ise cov erage, we are in terested in t he geometric and top ological c haracterization of a class ifier rather than its explicit functional 3 F or more on linear class ifi er s, see [16], Chapter 9 of [13], and Cha pter 4 of [5]. 6 form. A binary line ar classifier is a classifier of the f orm P = { M , N , R } where R is empt y and M , N are the op en and closed halfspaces, resp ectiv ely . Figure 4 b elo w is an example of a binary linear classifier. M N x 1 x 2 Figure 4: Example Binary Linear Classifier Let us consider the co verage prop erties o f a binary linear classifie r. By inspection, it is apparen t that man y p oin ts ha v e non-zero co verage. Recall that the de cision b oundary of a classifier is the set of p oin ts that separate lab els. Put another w a y , the decision b oundary for classifie r P with lab el set L P is g iv en b y S L ∈ L P bd ( L ) as with the example accompan ying Figure 1 in Section 2 . F or the binary linear classifier, its decision b oundary is the h yp erplane b ordering the tw o lab els. Note that the decision b oundary actually b elongs to one to the t w o lab els, since one lab el is a n op en half space and the other is a closed halfspace. As a result, the classifier has zero cov erage at eac h p oin t on the decision b oundary . Observ e that the set of points o n whic h the binary linear classifier has zero co ve ra g e is Leb es gue n ull. Moreov er, since the decision b oundary is the b oundary of an op en set, it is no where dense, implying that the decision b oundary is meagre. Let us in tro duce a refinemen t of the binary linear classifier by moving the decision b oundary from the feat ure space to the refinemen t set. A r efine d li n e ar cl a ssifier is a classifier of the form P = { M , N , R } where R is a hyperplane and M , N are the op en ha lfs paces ab o ve a n d b elo w R . An example is illustrated with Figure 5 b elo w. M N x 1 x 2 Figure 5: Example Refined Linear Classifier By mov ing from the binary linear classifier to the refined linear classifier, the classifier no longer has zero co v erage a t an y p oin t in the feature space. Moreov er, Theorem 4.1 7 demonstrates that the refined linear classifie r has infinite co verage at ev ery p oin t in the feature space. Theorem 4.1. If P is a r efine d l i n e ar c l a ssifier, then P has infi nite p oin t wise c over age. Pr o o f. Supp ose P = { M , N , R } is a refined linear classifier with L P = { M , N } where M , N are op en ha lf s paces a nd R is a h yp erplane. Let x ∈ S L P . Let O be a line con taining x and orthogonal to R . Without loss of generality , w e ma y assume that O ∩ R = 0 , i.e., the origin. Let O + = { y ∈ O |k y k > k x k} ∩ P ( x ). Let o 1 , o 2 , . . . b e a sequence of p oin ts on O + that are increas ingly far from x , i.e., k o i k < k o i +1 k . Let α < k x k . D e fine A i = B ( o i , r i ), where r i = k o i − x k + α . Observ e that A i ⊂ P ( x ), since r i < k o i k . Since x ∈ A i , i ≥ 1, eac h A i is an anc hor f o r x . Giv en that ( r i ) i ≥ 1 is an increasing sequence, w e ha ve that C P ( x ) = ∞ . 5 Infinite P o i nt wise Co v erage In this section, w e prov e the in ve rse of Theorem 4.1: if a non-trivial classifier has infinite p oin twise co v erage, then it is a refined linear classifier. T o attain this result, w e first pro v e the f ollo wing three lemmas. The first is a geometric prop ert y of an un b ounded sequence of balls with at least a single p oin t in common. The second applies the first lemma to the case of infinite cov erage at a p oin t in t he feat ur e space to imply prop erties ab out the shap e and size of the p oin t’s lab el. Finally , the third pro ves that if a classifier has infinite p oin t wise co v erage then it can ha v e at most t w o lab els in the lab el set. Lemma 5.1. If x ∈ R n and ( B i ) i ≥ 1 is an unb ounde d se quenc e of b al ls with e ach c ontaining x , B = S i ≥ 1 B i c ontains a n op e n halfsp ac e H such that x ∈ bd ( H ) . Pr o o f (with Pa ul L arson). Let x ∈ R n and B n = B ( q n , r n ) b e a ball cen tered at q n with radius r n > n suc h that x ∈ B n , n ≥ 1. Without loss of generalit y , let us assume that x = 0 , the origin in R n , and that ( r n ) n ≥ 1 is strictly increasing. F or eac h n ≥ 1, let s n to b e the unique p oin t on the line b etw een 0 , q n suc h that k s n k = 1. Since the unit sphere S n ⊂ R n is compact, let us fix s ∗ ∈ S n and supp ose s n → s ∗ . By rotating space, w e ma y assume that s ∗ = (1 , 0 , . . . , 0). Let H = { ( x 1 , x 2 , . . . , x n ) | x 1 > 0 } and θ p is the angle p 0 s ∗ for p ∈ H . Observ e that H is an op en halfspace and cos θ p > 0 for p ∈ H . While 0 / ∈ H , w e ha ve that 0 ∈ bd ( H ). W e w an t to sho w that H ⊂ B = S i ≥ 1 B i . Let d n = r n − k q n k . If d n is unbounded, then B = R n . Clearly , H ⊂ B . Otherwise, supp ose d n is b ounded. Since r n is increasing and un b ounded, k q n k increases to infinit y . F or p ∈ H , it is sufficien t to sho w that k q n k > k q n − p k , since B n con tains 0 , i.e., k q n k < r n . This distance condition is equiv alent to k p k < 2 k q n k cos θ n,p , where θ n,p is the angle p 0 s n . Since k q n k increases to infinity and θ n,p → θ p > 0 , w e can find a sufficien tly large n suc h that the distance condition holds. F or suc h an n , p ∈ B n ⊂ B . Lemma 5.1 b elongs to a family o f results connecting the structure o f spaces to the prop- erties of un b ounded sequences of balls [1]. Lemma 5.2. If P is a classifier such that the c ov er age of P at x is infi nite, then P ( x ) c ontains a n op en halfs p ac e H such that x ∈ bd ( H ) . 8 Pr o o f. Supp ose P is a classifier and the co verage of P at x is infinite. Then there exists a sequence of anc hors for x ( A i ) i ≥ 1 with un b ounded co v erage. Let A = S i ≥ 1 A i . Observ e that A ⊂ P ( x ). By Lemma 5 .1 , H ⊂ A , where H is an op en halfspace. Hence, H ⊂ P ( x ). Lemma 5.3. A cl a ssifier with infinite p ointwise c over age c an h ave at most two lab els in the lab el se t . Pr o o f. Let P b e a classifier with infinite global co v erage. Let x, y b e suc h tha t P ( x ) , P ( y ) ∈ L P and P ( x ) 6 = P ( y ). By Lemma 5.2, there exists op en halfspaces H x ⊂ P ( x ) and H y ⊂ P ( y ). Observ e that since P ( x ) , P ( y ) are disjoin t, bd ( H x ) , bd ( H y ) m ust b e pa r allel; otherwise, H x ∪ H y is non-empt y . F or con tradiction, supp os e there is a z / ∈ P ( x ) ∪ P ( y ) where P ( z ) ∈ L P . Applying Lemma 5.2 once more, w e obtain that there is an open halfspace H z ⊂ P ( z ). Th us, we ha ve R n con tains three disjoint op en halfspaces H x , H y , H z . Then bd ( H z ) mus t b e para lle l to bd ( H x ) , bd ( H y ); otherwise , the intersec tion of H z with eac h of H x , H y is non- empt y . O b serv e that the halfspace to one side of bd ( H z ) has a no n - e mpty inters ection with H x , while the other side has a non-empty interse ction with H y . ( →← ) . Observ e that a classifier with a single lab el can ha ve infinite p oin t wise co v erage; how ev er, a t r ivial classifie r do es not necessarily hav e infinite p o in tw ise co verage. W e provide a n example o f suc h a classifier in Section 6. With that b eing said, w e kno w that there is a unique ordinary trivial classifier, namely { R n , ∅} , and this classifier has infinite p oin t wise co v erage. With these lemmas in hand, w e may no w turn t o the main pro of in this section. Theorem 5.4. I f P is a non-trivial classifier with infinite p ointwise c ov e r age, then P is a r efine d line ar classifier. Pr o o f. Let P b e a no n-trivial classifier with infinite p oin twis e cov erage. Let x ∈ P ( x ) ∈ L P . Since P has infinite p oin twis e cov erage, the co v erage o f P at x is infinite. By Lemma 5.2 , H x is an op en halfspace such that H x ⊂ P ( x ) and x ∈ bd ( H x ). Since P is non-trivial, there is a y / ∈ P ( x ) where P ( y ) ∈ L P . By Lemma 5.2, H y ⊂ P ( y ) is an op en halfspace with y ∈ bd ( H y ). Observ e that since P is a partition, P ( x ) ∩ P ( y ) = ∅ . By Lemma 5.3, w e ha ve that L = { P ( x ) , P ( y ) } . Moreo v er, since P ( x ) , P ( y ) are disjoin t , bd ( H x ) , bd ( H y ) m ust b e parallel; otherwise, H x ∪ H y is non-empty . Let us define H P ( x ) = S z ∈ P ( x ) H z , where H z refers t o the op en halfspace generated b y applying Lemma 5.2 to z . W e claim that H P ( x ) = P ( x ). On the one hand, supp ose w ∈ H P ( x ) . Then, for some z ∈ P ( x ), w ∈ H z . Since H z ⊂ P ( x ), by construction, w ∈ P ( x ). On the other hand, supp ose instead that w ∈ P ( x ). Since P has infinite p oin t wise co verage, P has infinite cov erage at w . Let A b e an anc hor for w . Let us define b oth O w as the line containing w and orthogonal to bd ( H y ) and ˆ y = O w ∩ bd ( H y ) as the single p oin t common to b oth O w and the hy p erplane bd ( H y ). Cho ose a p oin t ˆ w from A ∩ O w where k w − ˆ y k > k ˆ w − ˆ y k . Applying Lemma 5.2 to ˆ w , w e obtain an op en ha lfs pace H ˆ w ⊂ P ( x ) where bd ( H ˆ w ) is parallel to bd ( H y ). By construction, w ∈ H ˆ w ⊂ H P ( x ) . W e hav e established that P ( x ) is the countable union o f op en halfspaces. W e further claim tha t P ( x ) is an op en halfspace. Let us first note that P ( x ) is op en since it is the union of op en sets. F or each z ∈ P ( x ), bd ( H z ) mus t b e parallel to bd ( H y ), implying bd ( H z ) 9 is parallel to bd ( H z ′ ) for z , z ′ ∈ P ( x ). Then H z ∪ H z ′ is either H z or H z ′ , i.e., a halfspace, and so fo r t h. With P ( x ) being an op en halfspace, we also hav e that P ( y ) is an op en halfspace by symmetry . Recall that R is meagre by a ssumption and that bd ( P ( x )) ∪ bd ( P ( y )) ⊂ R . Then R contains no non-trivial op en set. F or con tra dic tion, let us supp ose tha t bd ( P ( x )) 6 = bd ( P ( y )). Let O b e a line orthog onal t o b oth bd ( P ( x )) , bd ( P ( y )); suc h a line exists, b ecause bd ( P ( x )) , bd ( P ( y )) are parallel. Let ¯ x = O ∩ bd ( P ( x )) and ¯ y = O ∩ bd ( P ( y )). Define α < k ¯ x − ¯ y k and r = ¯ x + ¯ y 2 . Then B ( r , α ) ⊂ R , but R is con tains no non-trivial op en set. ( →← ). W e obtain P = { P ( x ) , P ( y ) , R } where P ( x ) , P ( y ) are op en halfspaces and R is the closed meagre hy p erplane separating the halfspaces. Hence, P is a refined linear classifier. Theorem 5.4 provides a link b et ween the p oin t wise co ve rag e of a classifier and the g eom- etry and top o logy of its lab els. In particular, if a classifier has infinite p oin twis e co v erage, then its la bels consist of tw o op en halfspaces. This result accords with our in tuitions ab out infinite co v erage and the curv ature of the decision b oundary . Namely , if the decision b ound- ary is curv ed, then on one side of the b oundary , for some p oin t in that lab el, anc hors will b e b ounded in size. Moreov er, along with Theorem 4.1, Theorem 5.4 implies the primary result of this pap er: Theorem 3 .1 , a characterization of the most explainable collection of classifiers. An ob vious corolla r y o f this resu lt is that no ordinary classifie r has infinite p oin twise co v erage. In Section 6, w e explore the collection of classifiers t ha t can b e refined to the refined linear classifier, i.e., to a classifier with infinite p oin t wise co ve rag e . 6 Refining Ord i nary Classifie rs In Sections 4 and 5, we establishe d that the refined linear classifier is uniquely the most explainable classifier. While t his is an inte resting prop ert y of the p oin tw ise cov erage f r a m e- w ork, in isolation, intere st in this result is limited to strictly theoretical concerns . Recall that only o rdinary classifiers are found “ in the wild”. T o this end, in Section 4, w e illustrat e d that a refined linear classifier is a refinemen t of a binary linear classifie r, a typical o rdinary classifier. In this section, w e identify the collection of ordinar y classifiers t ha t can b e refined to a refined linear classifier: the generalized binary linear classifiers. It is worth t a k ing a momen t t o reflect on what is a refinemen t of a classifier. In the examples prov ided th us far, refinemen ts ha ve b een used to increase p oin tw ise co v erage of a classifier b y remov ing edge cases from the feature space, particularly those along the decision b oundary b et w een the lab els . F or example, while the refined linear classifier has infinite p oin twise co v erage, the bina r y linear classifier do es not; its edge cases hav e zero cov erage. By mo ving to the refined mo del in the case of the linear classifier, w e are able to measure and agg re ga te the explainability of the mo del without inte rf erence f r om edge cases. It is not the case, ho w eve r, that all refineme nts impro ve or ev en prese rve p oin t wise co v erage. In fact, f or a n y classifier, there exists a refinemen t that has zero p oin t wise co v erage. Recall that a classifier has zero p oin twise co v erage if the suprem um of cov erage ov er all p oin ts in the feature space is zero. T o see this, let P b e a classifier on R n . Observ e that Q n is 10 meagre, Leb esgue null, and dense in R n , i.e., Q n = R n . Let P ′ b e a refinemen t of P by mo ving Q n from the feature space o f P to the refinemen t set. Since Q n is dense in R n , an op en ball con ta ining any part icular p oin t x in the feature space will also con t ain a p oin t in Q n ⊂ R . Hence, there are no anc hor s fo r x , so P ′ has zero point wise co v erage. Just a s there are unexplainable o rdinary classifiers, there are unexplainable unexplainable refined classifiers. Let us in tro duce some helpful terminology for refinemen ts. A lab el is called ne gli g ible if it is b oth meagre a nd Leb e sgue n ull. W e say that a r efinemen t P ′ of classifier P is elimin at ive if L P ′ is a strict subset of L P . Observ e that a classifier P has an eliminativ e refinemen t just in case P contains a negligible lab el. Let us supp ose tha t P = { M , N , R } is a refined linear classifier where M , N are o p en halfspaces and R is their separating h yp erplane and Q is a n or dinar y classifier with no negligible lab els suc h that P is a refinemen t of Q . Whic h classifiers satisfy the conditions of Q ? Since Q has no neglig ible lab els, P is not an eliminativ e refinemen t. Then there exists A ∈ L Q suc h that M ⊂ A and B ∈ L Q suc h that N ⊂ B . Moreo ve r, since Q is ordinary , its refinemen t set is empt y , implying A ∪ B = M ∪ N ∪ R . It ma y b e that R ⊂ A or R ⊂ B , in whic h case Q is a binary linear classifier. Additionally , it may b e the case that some of R b elongs to A and the rest of R b elongs to B . By remo ving the constraint that Q has no negligible lab els, we extend the collection of o rdinary classifiers to include those which partition R in to arbitrarily man y , ev en coun tably many , lab els . W e refer to this collection as the gener ali z e d binary line ar classifiers . F rom the p erspective of p oin twis e co v erag e, a ll generalized binary linear classifiers are equiv alen t: they can b e r efined to the refined linear classifier. There is a sense in which the binary linear classifie r is more natura l than the o t h er generalized binary linear classifie rs; ho w ev er, that is outside of the scop e o f the p oin twis e cov erage framew ork and is pres ently left to heuristic. Let us conclude this section by not in g that a generalized binary linear classifier is equiv alent t o a binary linear classifier up to null sets, since the decision b oundary is a Leb esgue null set. 7 Conclus ion By in tro ducing a formal framew ork fo r classifiers and the to pological notion of p oin tw ise co v erage, we are able to express what is meant by the most explainable classifier, infinite p oin twise cov erage, and identify the unique collection o f class ifiers with this prop ert y , the refined linear classifiers. Moreo v er, up to n ull sets, only one classifier found “in the wild” can b e refined to a refined linear classifier: a binary linear classifier. This result a ccords with our in tuitions ab out the simplicit y , utility , and explainabilit y of the binary linear classifier. References [1] Bandyop adhy a y, P., and Lin, B.-L. Some prop erties r e lated to nested sequence of ball in bana c h sap ces. T aiwanese J. Math. 5 , 1 (03 2001), 1 9 –34. 11 [2] Gilpin, L. H., Bau, D., Yuan, B . Z. , Bajw a, A., Specte r, M ., and Kagal, L. Explaining explanations: An o v erview of in terpretability of mac hine learning. In 2018 IEEE 5th I n ternational Confer enc e on Data Scienc e and A dvanc e d Analytics (DSAA) (Oct 2018), pp. 80–8 9. [3] Guidotti, R ., Monreale, A., Ruggieri, S., Pedre schi, D., Turini, F., and Giannotti, F. Lo cal rule-ba sed explanations of black b ox decision systems. ArXiv abs/1805.108 2 0 (2018). [4] Guidotti, R ., M onreale, A., Ruggieri, S., Turini, F., Giannotti, F., and Pedreschi, D. A surv ey o f metho ds for explaining black b ox mo dels. ACM C omput. Surv. 51 , 5 (Aug. 2018 ), 93:1–9 3:42. [5] Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistic al L e arn- ing . Springer Series in Statistics. Springer New Y o rk Inc., New Y ork, NY, USA, 2 001. [6] Kechris, A. S. C l a ssic al Desc ri p tive Set Th e ory . Springer-V erlag , 199 5 . [7] Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., and Detyniecki, M. The dangers of p ost-ho c in terpretabilit y: Unjustified coun terfactual explanations. In IJCAI (20 19). [8] Molnar, C. Interpr etable Machine L e arning: A Guide for Making Black Box Mo de l s Explainable. 2018. [9] Munkres, J. R. T op olo gy , 2nd ed. ed. Prentice Hall, Inc, 2000. [10] Ribeiro, M. T., Singh, S., and Guestrin, C. ”wh y should i trust y ou?” : Ex - plaining the predictions of an y classifier. In Pr o c e e dings of the 22n d ACM SIGKD D International Confer enc e o n Know le dge Disc overy and Da ta Mining (New Y ork, NY, USA, 2016 ) , KD D ’16, A CM, pp. 113 5–1144. [11] Ribeiro, M. T ., Singh, S., and Guestrin, C. Anc hors: High-precision mo del- agnostic explanations, 2018. [12] Rosenbla tt, F. The p erceptron: A probabilistic mo del for informa t ion storage and organization in the bra in. Ps ych o l o gic al R ev iew (1958), 65 – 386. [13] Shalev-Shw ar tz, S., and Ben-Da vid , S. Understanding Machine L e arning: F r o m The ory to A lgorithms . Cambridge Univ ersit y Press, New Y ork, NY, USA, 2014. [14] Singh, S. , Tulio Ribeiro, M., and Guestrin, C. Programs as Black-Bo x Expla- nations. arXiv e-pri n ts (Nov. 20 1 6). [15] Sriv ast a v a, S. M. A Course on Bor el Se ts . Spring er- V erlag, 1998. [16] Ustun, B., and R udin, C. Metho ds a nd Mo dels for In terpretable Linear Class ifica- tion. arXiv e- p rints (Ma y 2014). [17] V apnik, V. N. Statistic al L e arni ng The ory . Wiley-In terscience, 19 9 8. 12 [18] Yin, M., Wor t man V a ughan, J., and W allach, H. Un derstanding the effect of accuracy on trust in machine learning mo dels. In Pr o c e e dings of the 2 019 CHI Con fer- enc e o n Human F actors in Computing Systems (New Y o rk, NY, USA, 2019 ), CHI ’19, A CM, pp. 279:1–27 9:12. 13