Evaluation for Uncertain Image Classification and Segmentation

Ev aluat ion of Uncer tain Ima ge Cl assificat ion and Seg m en tati on Algor ithms Arnaud Martin a , Hic ham Laanaya ab and Andreas Arnold-Bos a a ENSIET A E3I2 EA3876, 2 rue F ran¸ cois V ern y , 29806 Brest Cedex 09, F rance b F acult ´ e des sciences de Rabat, Aven ue Ibn Batouta, B.P . 1014 Rabat, Moro cco Eac h year, numerous segmentatio n and classification algorithms are in ven ted or reused to solve prob lems where mac hine vision is n eeded. Generally , the efficiency of these algorithms is compared against the results given by one or many h uman exp erts. How ever, in many situatio ns, the lo cation of the real b oundaries of the ob ject as wel l a s their cla sses are not kno wn with ce rtaint y by the human exp erts. Moreo ve r, only one asp ect of t h e segmen tation and classification problem is generally ev aluated. In our ev aluation metho d, w e take into accoun t b oth the classification and segmen tation results as we ll as t h e level o f certaint y given b y the exp erts. As a concrete example of our method , w e ev aluate an automatic seabed characteri zation algorithm based on sonar images. 1. INTR ODUCTION Image classification and segmen tation ar e tw o fundamen tal problems in imag e analysis. Seg- men ting an image cons ists in dividing the imag e int o homogeneo us zones delimited by b oundaries so as to separ ate the differen t entities visible in the image. Cla ssification consists in lab eling the v ar ious comp onents visible in an image. A grea t deal o f segmentation and classification methods hav e b een prop o s ed in the las t thirt y y ears [3]; enum erating them all is no t the purp ose of our pap er. How ever, an imp orta n t question to so lve is how to b enchmark these metho ds and ev aluate their robustness with resp ect to a given re al-life application. A t ypical example of the use o f classifica tion and segmen tation is encountered in satellite or sonar imaging, where a n impo r tant use of the data is to classify the types of soils pre sent in the images , for instance to build maps. As the amount of images gathered during a missio n is impo rtant, automatic reco g nition alg orithms ca n relieve human op erators . Since the sw ath of th e sensor is wide, many types of s oils can b e en- countered wit hin a single ima ge, a nd the cla ssi- fication m ust be done on a loca l neigh bo rho o d. This neighborho o d can b e either limited to a sin- gle pixel, or often to a small tile o f e.g. 1 6 × 16 or 3 2 × 32 pixels taken as the unit for the classi- fication algorithm. The b oundaries be tw een the different patc hes corresp onding to a c a tegory of soil ar e a for m of segmentation, which is here an implicit byproduct o f the c lassification. In other applications, s egmentation can come first so as to isolate entities which will be lab eled later. A difficult y raised in these applications is the lack of ground truth which could b e used to ev al- uate the res ult of the classifica tion. The real re f- erence classes must b e estimated b y human ex- per ts from the data themselves. How ev er, the im- ages ar e difficult to read since they are corrupted by man y phenomena and the estimation of the classes by the h uman exp ert will b e hig hly sub jec- tive and with a v a r ying level of uncertaint y . In the case of the automatic seab ed classifica tion, which we will use as our reference exa mple throughout this paper , images ar e esp ecially hard to interpret due to ma n y imp erfections [2]. T o reco ns truct the image, a h uge num ber of pa rameters (geometr y of the device, co ordinates of the ship, mov ements o f the sonar , etc.) are taken into a ccount, but these 1 2 A. Martin et al. data are p o lluted with a lar ge a mount of sensor noise. Plus, other phenomena suc h as multipath signal pr opagation (caus e d b y refle c tio n either on the b ottom or the surface), speckle, and the pr es- ence of fauna and flora ( e.g. shadows o f fishes on the sea bo tto m), will a ll augment the difficulty of int erpretatio n of the imag e . Conseque ntly , differ- ent exp erts can prop ose different cla ssifications of the image. Thus, in or der to ev aluate a utomatic classification, we must take into account this dif- ference a nd the uncer taint y of ea ch exp ert. Fig- ure 1 exhibits the differences betw een the in ter- pretation and the ce rtaint y o f two sonar experts trying to differentiate the t yp e of sedimen t (ro ck, cobbles, sa nd, ripple, silt) o r sha dow when the in- formation is invisible (each color c o rresp ond to a kind of s e diment a nd the asso ciated certaint y of the exp ert for this sediment expres sed in term of sure, mo dera tely sure a nd not sure). Figure 1. Segmentation g iven b y t wo exp erts. W e prop ose in this article a ne w approach for image classification and se g ment ation ta k ing into account the information giving b y multiple ex- per ts a nd the certaint y of the given information. Classical ev aluations of the classifica tion and seg- men tation do not take int o acc o unt the uncer - tain and imprecise lab e ls in the reference image provided by an e x per t. W e think that w e have to conside r these kind of la be ls in our ev alua- tion approa ch. In section 2 w e sho w ho w to in- tegrate the exper t certain ty in confusion matrix and so to deduce a go o d class ification r ate and error clas sification rate. Moreo ver, our thesis is that global image classification ev aluatio n must be made not only by ev aluating the classificatio n on considered units (with the confusion matrix) but also by ev a luating, at the sa me time, the in- duced segmentation. In section 3, we prop ose tw o new dista nc e -based meas ures in o rder to ev aluate well and mis-segmented pixels by taking into ac- count b oth the lo ca tion of the bor ders and the exp ert cer taint y . Note that a nother impo rtant criterion to ev aluate class ification/seg mentation approaches is the ev aluation of the co mplex it y of the algo rithms [1], but we do not co ns ider it in this pap er. Finally , our ev a lua tion is illustr ated in section 4 on real sonar images acquired in a real, uncertain environment. 2. CLASSIFICA TION EV AL UA TION T r aditional classificatio n systems can usua lly be describ ed as a three- tiered pro cess. First, sig- nificant features ar e ex tracted from the images to class ify . These features a re widely differe n t, depe nding on the application; they are gener a lly describ ed using a sma ll set o f abstract numeri- cal meas ures. F or example, used features may b e the lo cal luminance, the texture (described with measures such as the entropy , the co-o currence matrices, etc), the contours (describ ed with their length, their orientation, their re lative po sition to other contours, etc) [3]. Most of the time, a sec- ond stage is necess ary to re duce these features, bec ause there ar e too numerous. In the third stage of the algorithm, the numerical descrip- tors are fed to clas s ification a lgorithms, which are application-indep endent, such as Supp ort V ec- tor Mac hine [4 ,5,6], neural net w orks [2 ,6,7,8], k - Ev aluation of Uncertain Imag e Cla ssification and Segmentation Algorithms 3 nearest neighbors [9], etc. The classifica tion al- gorithms will decide, depending on their entries, which is the class of the imag e. Hence, we hav e to ev aluate these cla ssification algorithms in order to c o mpare their robustness in a given applicatio n. The cla ssical approa ch is based o n the confusion matr ix and do e s no t take int o acco un t uncertain labels. W e prop os e here a new confusion matrix and go o d clas sification and erro r r ates taking into account these kind of lab els a nd also the inhomogeneous units defined forwards. The pr o p o sed metho d of ev a lua tion in this sec- tion, can be applied for the ev alua tion of a classi- fication algo rithm in every domain where uncer- tain la bels are pr ovided. W e do not co nsider here the problem o f the learning on uncer tain and im- precise labe ls [10,11,12]: the classification can be made b y this kind of algo rithms or o thers. 2.1. Classical E v aluation The results of one image class ification can be observed and visually compar ed to the realit y . But in order to ev aluate a classifica tion algorithm, many different configurations and tests must be considered. Classification algor ithms can yield very v ariable r esults dep ending o n the s a mple. Generally class ification algor ithms ev alua tion is conducted b y the confusion matrix . Confusion matrix is comp osed by the num ber cm ij of ele- men ts from the class i c lassified in the class j . In order to obtained rates, with one more easier to int erpret, we can normalize this confusion matrix by: N cm ij = cm ij P N j =1 cm ij = cm ij N i , (1) with N the n umber of co nsidered clas s es and N i the num ber of ele men t from the true class i . F rom this nor malized confusion ma trix a g o od classifi- cation rate vector can b e written a s: GC R i = N cm ii , (2) and an error classifica tio n rate vector a s : E C R i = 1 2   N X j =1 ,j 6 = i N cm ij + N X i =1 ,i 6 = j N cm ij N − 1   . (3) This err or cla ssification rate is the mean o f the t wo errors corres po nding res pec tively to the el- ement s fro m a g iven clas s i falsely cla ssified as elements of a nother class (first ter m), and to the elements classified in a g iven class j but b eing from another class i (second ter m). These error s are a lso called err ors of fir st and seco nd kind. W e do not hav e to normalize the first term beca use of the normaliza tion of the confusion matrix on the rows, but the second ter m must b e no rmalized by the num ber of rows minus one (b ecause of the N cm ii term corre s po nding to the g o o d classifica- tion). Note tha t other err or rates can be defined (see e.g. [10]). W e hav e seen that image classifica tion algo- rithms ev aluation must b e ma de not only on one image but on the whole image databa se. As a trivial consequence, w e ha ve to consider a no n- normalized confusion matrix on eac h image and normalize the sum of the matrix confusion o n all images of the da ta base. 2.2. Ev aluation with expert information Consider a gener a l case wher e information is given b y the exper t o n each pixel and the clas - sification algorithm is made o n an unit of n × n pixels. In such a case, if a n × n tile is consid- ered, mor e than one cla s s can b e present (we call it pa tc h-worked tile or inho mo geneous unit), a nd the class ific a tion algorithm can find only one of these clas s . In o rder to take into account the last example, we cons ider that if the class ific a tion al- gorithm finds one of these c la sses on the tile, the algorithm is right in the prop ortion of this c la ss found in the n × n tile a nd it is fals e in the pro - po rtion of the other classes in the tile. F o r in- stance, imagine the case where the ex per t con- siders a 16 × 16 tile and declares that 156 g iven pixels b elo ng to cla ss 1, a nd 100 other pixels b e- long to clas s 3. If the classificatio n algo rithm finds the tile belo ngs to class 1, the confusion matrix will b e computed b y cm 11 = cm 11 + 156 / 25 6 and cm 31 = cm 31 + 100 / 25 6 . Hence the confusio n ma- trix is no t compo sed of integer num b e rs and N i is also not integer, but the sums o f column are still int eger. Now cons ider the case where the exp ert gives the class with a certaint y grade . F or instance, 4 A. Martin et al. the op erator can b e mo der ately sure o f his choice when he lab els o ne par t of the image as belong - ing to a certa in cla ss, and b e totally doubtful on another part of the image. In our classification ev a luation we must not take these tw o r eferences equally . Indeed, classical co nfusion matrices im- ply that the reality is perfectly kno wn; this, un- fortunately , is no t the case in many real appli- cations. W e prop ose to represent this difference of information by different weigh ts cor resp onding to the different certaint y g r ades that ar e co nsid- ered. F o r exa mple, if three grades o f certaint y (sure, mo der ately sure and not sure) are con- sidered, we can pr ovide r esp ectively the weight s: 2/3, 1/2 and 1 /3. In the confusion ma trix, such weigh ts co uld be in tegrated easily in the g eneral sum. If one exp ert lab els a tile as be lo nging to class 1, with a mo dera te certa in ty , and if the clas- sification algor ithm finds the cla ss 1, considering the previous g iven weigh ts, the c onfusion ma trix will be updated suc h a s: cm 11 = cm 11 + 1 / 2 . If the cla ssification algorithm finds the class 2 on the cons idered tile, the confusion matrix beco mes cm 12 = cm 12 + 1 / 2. Hence the sums of column are not integer anymore. In or der to take into acco unt the refere nce d im- ages provided by differen t exp e r ts, we can com- pare the classified image with all the exp ert- referenced images. Hence we o btain as many con- fusion matrice s as e x per ts, and we can simply combine them by a ddition. By the simple fact that w e add the non- normalized confusion matrices, we weigh t the ob- tained results by the image size o r the consider ed unit num ber. Consequently , in order to o btain r ates, we can normalize the o btained confusio n matrix with equation (1) a nd calcula te the go o d class ification rate vector with equation (2) and the er ror classi- fication rate vector with equation (3). O f cours e these r ates are not pe r centages anymore. F or in- stance, the go o d classifica tio n ra te is no t p ercent- age of w ell classified units an ymore, b ecause the weigh ts given by the inhomogeneous units or by exp ert certaint y a r e rational. In conclus io n of this s e ction: the interest of these newly obtained confusion matrix, g o o d c las- sification r a te and er ror classification ra te is that, they give a goo d ev a luation of class ification tak- ing into a ccount the inhomogeneous units and un- certaint y of the exper ts. This approach can be applied in o ther applicatio ns than ima ge class ifi- cation, in fac t in ev ery domain where w e try to classify uncertaint y elemen ts. 3. SEGMENT A TION E V ALUA TION Segmentation can either b e obtained as a byproduct of the classification, a s shown abov e, or be used as the first step of an image pr o cess- ing pip eline. Ma ny metho ds o f image segmen- tation and edge detection hav e b een prop osed [14,15,13,16,17]. It is impo rtant to be able to benchmark these metho ds a nd to ev a luate their robustness; but to do that, meas ur es a re needed so as to have an ob jective means to judge the quality of the se gmentation. No p erfect measur e exists to day , and existing measures are not w ell satisfied, this is wh y we can imagine fusing the segmentation ev aluation approaches [18]. On the one hand the image cla ssification meth- o ds are ev aluated by the confusion matr ix. Go o d classification rates and error rates are usually cal- culated from this matrix . Note that in o rder to es - tablish the confusion matr ix, the r eal class o f the considered units of the images need to b e known. This gives only an ev aluation of the classification approach on considered units of the image, but do es no t give a n ev aluation of the pro duced seg- men tation. On the other ha nd, segmentation ev aluation cannot be made only by visual compar ison b e- t ween the initial image and the segmented image. Many ev aluation a pproaches hav e be en prop osed for image seg ment ation [1,16,19,20,21]. W e can consider t wo cases: we do not hav e any a pri- ori knowledge of the corr ect segmen tation, and we have an a priori k nowledge of the correct seg- men tation. In the first case, many effectiv eness measures based on intra-region uniformity , inter- region co n trast and r egion shap e have be en pro- po sed [1]. The second case implies to get refer- enced images. In a real application, exp erts must manually provide the image segmentation via a visual insp ection. [1] gives a r e v iew of usual dis - crepancy mea sures based on different dista nces Ev aluation of Uncertain Imag e Cla ssification and Segmentation Algorithms 5 (sometimes expressed in ter ms of pro bability) b e- t ween the segmented-pixel and the re ferenced- pixel. Most of the time, only a measure of how many pixels are mis-segmented is g iven. W e, o n the contrary , prop ose in this article a combined study of one w ell-segmented pixel measure and a mis- segmented pixel mea sure. Indeed, most of the time, when a pixel is not mis-s egmented, it is not necessary w ell-segmented either. As a c on- sequence, w e can hav e few mis-segmented pixels but also few well-segmen ted pixels, which means that the segmentation is not g o od ov erall. In order to calculate confusion ma trices we need the a priori knowledge o f the class for each pixel or at least for each considered unit of the image. Hence, expe r ts hav e to give referenced images, and we can consider to b e in the seco nd case of segmentation e v aluation that we desc rib ed ab ov e. Before pr e senting our metho d o f segmentation ev a luation, w e show how we can easily o btain a deducted segmentation from a n image classifica- tion ba sed on the class ification on tiles. Next, the prop osed segmentation ev aluation metho d is adapted to every ima g e segmen tation a nd can take in to acco un t imprecise lab els. 3.1. Deducted s egmenta tion Image c la ssification provides a n implicit image segmentation given by the difference of cla sses b e- t ween tw o adjace nt tiles. Hence a go o d image classification ev aluation should take this seg men- tation in to account as well. First of all, w e hav e to define the b ounda r y pixels g iven by the image classificatio n. W e pr o- po se here to use a very simple approa ch: we will take as b o unda ry pixels, the pixels whic h neigh- bo r another class on the rig h t o r/and on the b ot- tom. F or instance, on table 1 we give a dummy segmented image with tw o classes given by × and • . The cla ssification unit is here 4 × 4. The bo undary pixels are underlined. Many appro aches can b e co nsidered in o rder to obtain bo undaries without angula r p oints. W e can consider for instance an in terp olation b e- t ween the 4-co nnexity o r 8 - connexity po int s [22]. This is not the sub ject of this paper; the reader T a ble 1 Example of an obtained seg men tation on imag e with tw o clas s es given b y × and • . × × × × • • • • × × × × • • • • × × × × • • • • × × × × • • • • • • • • × × × × • • • • × × × × • • • • × × × × • • • • × × × × should keep in their mind tha t our seg ment ation ev a luation is genera l and can b e applied to all image segmentations given by b oundary pixels. 3.2. Segmentation ev aluation W e recall here that in our c ase, we hav e an a priori knowledge of the corr ect or a pproximately correct segmentation given by the exp er ts. In this case all ev aluation a pproaches ar e based on different distances (or probabilities) b etw een the segmented-pixel and the refere nc e d- pixel [1,2 3,24] and most of the time o nly one measure of mis- segmented pixel is given. W e think that it is not enough for a precise segmentation ev alua tio n if a pixel can b e no t mis-seg ment ed, and also no t well-segmen ted. As we mentioned b efore, we ca n hav e few mis-seg ment ed pixels o nly with few w ell- segmented pixels, and so the segmentation can- not be considered right. So we pr op ose a link ed study of t w o new measures : one w ell-segmented pixel measure a nd one mis-segmented pixel mea - sure. Moreover these tw o measures can take into account the uncertaint y of the exp er t on the p o- sition and on the existence of the b o undaries if this uncertaint y c a n be expressed as a weigh t. 3.2.1. Boundary go o d detection measure The well segmented pixel meas ure is a mea - sure of how the bo undary is well detected a nd the mis-segmented pixe l mea sure tries to quan- tify ho w many b ounda ries detected by the al- gorithm to b enchmark have no physical r eality . First, we s e a rch the minimal distance d f e betw een each bounda ry pixel f fo und by the alg orithm to 6 A. Martin et al. benchmark, and all the b o undary pix e ls e pro- vided by the exp ert. Hence the pixel e is a func- tion of f , and we should note it as e f , but in order to simplify notations, it is referred as e in the rest of pape r . W e take here an Euclidean dis- tance but any other dis tance can be envisaged. The certain t y weight of the pixel e g iven by the exp ert is noted a s W e . W e define a well-detection criteria vector by: D C f = exp( − ( d f e ∗ W e ) 2 ) ∗ W e . (4) This criteria gives a Gaussian- like distribution of weigh ts with a standard deviation giv en b y the certaint y weights as shown in figure 2. Figure 2. Distance w eight for the w ell-detection criteria. The b oundary g o o d detection mea sure is de- fined by the normalized w ell-detection criteria given by: W D C = P f D C f (max f ( D C f ) ∗ P e W e ) a . (5) The nor ma lization is made in o rder to obta in a measure defined betw een 0 a nd 1. How ever, in real applica tio ns, this criter ia remains small even for very go o d b oundar y de tectio n. So we take a = 1 / 6 in order to accentuate sma ll v alues. This cr iteria is not co mpletely sa tisfying b e- cause it only takes into account the distanc e from the found b ounda ry to the contour pro vided by the exp ert. Ho wev er, the reference b oundary a lso has a lo cal dir e ction which is another informa- tion we w an t to use. A boundar y found by the algorithm can come a cross a b oundar y giv en by the exp ert ortho g onally: in this case some pixels from the found b oundar y are very near (in terms of distance) to pixels fro m the g iven bo undary but we do not wan t say that is a go o d detection. W e prop ose tw o w ays to consider the direction of bo undaries. In the first one, w e coun t, for a given pixel f of the found b oundary , how man y pixels from the found b oundary a r e linked b y the minimal dis- tance to the same pix el e of the refer enced b ound- ary . This num b er is noted n ef , e.g. o n figure 3 we have n ef = 3 for three different f . W e redefine the well-detection b oundary mea sure by: W D C = P f D C f /n ef (max f ( D C f /n ef ) ∗ P e W e ) a . (6) Figure 3 . Ex ample of n ef for thre e giv en f , the found b oundary is r epresented b y green s q uare and the refer e nced b ounda ry by black line. The pro blem is th at the n um ber n ef do es not necessarily repr e s ent a num ber o f pixels on the same b oundary and takes well into acco unt only the orthogo na l direction. Ho wev er th is measur e gives the b est ev aluation o f the pro p o rtion of the found b oundaries. The second metho d is based on the idea that the lo cal direction of the bo unda ry should also b e taken into acco unt : the dire c tion of the detected bo undary and the direction of the b oundar y g iven by the exp er t s hould b e the same. Now, how do es one c o mpute the direction of the b oundary? Let I r denote the r eference b oundar y imag e given b y the exper t. I r ( i, j ) = 0 if no b oundary is detected at pixel ( i, j ); I r ( i, j ) = W e otherwise, wher e W e is the weight of the pixel b o unda ry e a t lo catio n ( i, j ) given by the e x per t. Image I r can b e seen as a discrete 2-D function on which the gradient − → g r = [ ∂ I r /∂ x ; ∂ I r /∂ y ] can b e computed. The Ev aluation of Uncertain Imag e Cla ssification and Segmentation Algorithms 7 gradient has the pr o p e rty to b e nor mal to iso- v alue s lines of I r and will therefor e be normal to the b oundar ies given by the exp ert. Similarly , one can also c o mpute the g radient − → g s of the found bo undary image. Then, a meas ure of corres p o n- dence betw een the directions at pixel ( i, j ) can b e given b y the absolute v alue of the norma lized dot pro duct betw een the tw o gr adients vectors 1 : B D = | − → g r . − → g s | || − → g r || . || − → g s || . (7) How ev er, as I r is mostly filled with zeros, the gradient will have a negligible v a lue at most lo- cations. The farther a pixel is from a boundary given b y the op erato r, the low er the gradient at that pixel will b e, thus yielding a huge impreci- sion o n the lo cal dire c tion o f the image. T o solve this problem, we used the Gr a dient V e c tor Flow (GVF), first in tro duced b y Xu and P r ince [25]. F o r a bo unda ry ima ge I , the GVF is a vector field − → f = [ u ( x, y ); v ( x, y )] that is computed iteratively so as to minimize the following cost function over all the b oundary image: U = Z Z  µ. ( u 2 x + u 2 y + v 2 x + v 2 y ) + . . . + || − → g || 2 || − → g − − → f || 2  .dx.dy . (8) where µ is a tunable w eight, v ariables in indices denote partial deriv ation with resp ect to that v ar iable, and − → g is the gr adient of the imag e as defined prev io usly . This cost function was de- vised so that on bo undaries, where the gradien t is high ( || − → g || → ∞ ) the energy rema ins bounded: || − → g − − → f || must tend to zero if one wishes the inte- grand to b e minimized. Thus, on b oundaries, t he GVF is e qu al to the gr adient field . On the other hand, for pixels far aw a y fro m an y b oundary , the gradient will tend tow ard zero , and the integrand will b e driven by the term µ. ( u 2 x + u 2 y + v 2 x + v 2 y ). T o minimize it, the pa rtial deriv ativ es o f the v ector field − → f must b e null, which means that t he GVF extends the gr adient by c ontinuity to zones wher e it would normal ly b e ne gligible . The GVF is co m- puted b o th for the reference image and the image 1 The no tation “.” for m ultiplication is a term b y term multiplication of the tw o matrices. obtained through s egmentation. The measure of corres p o ndenc e b etw een the b ounda ry directio ns will b e similar to equatio n (7): B D = | − → f r . − → f s | || − → g r || . || − → g s || . (9) On figure 4, note that the gr adient is only strong on edges, whereas the GVF is strong ev- erywhere, thus enabling the lo ca l directions to be seen. Figure 4 . Computing the dir ection o f the b o und- aries: gradient (top), GVF (b ottom). Hence, we can rede fine D C f in equation (6) by ( DC .B D ) f , so that we o btain a new measur e which takes in to account the lo cal direction of the found b oundaries. 8 A. Martin et al. 3.2.2. F alse detection b oundary measure The bo undary false detectio n mea sure is based on the same principle than the well-detected bo undary measure, but the Gaussian-like distri- bution of weigh ts must b e inversed. Hence we can defined a false detection criteria b y: F D C f = 1 − D C f /W e , (10) where the pixels f and e ar e link ed by the min- imal distance d f e . As a consequence, the false detection b oundar y measure can b e defined by the normalized false detection criteria b y: F D = 1 − exp  − P f ( F DC f ∗ n ef ) max f ( F DC f ∗ n ef ) ∗ P e W e  . (11) In o rder to take into acc ount the lo cal direc- tion of the found b oundaries as found with the GVF, we can r edefine D C f in equation (6) by ( F D C . (1 − B D )) f , so we o bta in another new false detection criteria . Here we hav e descr ibed the use of measures F D and W D C for one image classified by the algo- rithm a nd another image provided by o nly one exp ert. In or der to ev aluate ima ge segmentation algorithms on ma n y images we can use a w eighted sum of these b oth measures, taking into acco unt the image sizes, which ca n b e different for all con- sidered imag es. In conclusion of this section, we have describ ed t wo new mea s ures F D and W D C taking into ac - count the uncertaint y o f differ ent exp erts on the seen boundaries . W e ha ve to consider these t w o measures together . 4. ILLUSTRA TION W e present her e an illustra tion of our image classification and segmentation ev a lua tion on r e al sonar imag es. Indeed, underwater environmen t is a very uncertain environment a nd it is par- ticularly impo rtant to class ify seab ed for numer- ous applicatio ns s uch as Autonomous Underwater V ehicle navigation. In recent sonar works ( e.g. [26,27]), the classifica tion ev aluation is ma de only by visual compariso n o f o ne original image and the classified ima ge. That is not satisfying in order to corre c tly ev aluate imag e classification and seg men tation. First we present o ur database given by t wo different exp erts with differ ent cer- tainties. Then, one p ossible classifica tion ap- proach for sonar imag e is pr esented. Finally the automatic cla ssification and segmentation ob- tained b y this approa ch is ev aluated with our new ev a luation metho d. Note that this illustration is presented in order to show how our mea sures work on only o ne cla s- sifier. In order to ev aluate a clas sifier, we hav e to compare the results with another classifier or with other parametr iz a tion of the ev a lua ted c la ssifier. 4.1. Database Our da tabase contains 42 sonar imag es pro- vided by the GE SMA (Gro upe d’Etudes Sous - Marines de l’Atlan tique). Theses images were ob- tained with a Klein 540 0 latera l so nar with a r es- olution of 20 to 30 cm in azimuth and 3 cm in range. The sea-b ottom depth was b etw een 1 5 m and 40 m. The exp e rts hav e manually segment ed these images giv ing the kind of feature v isible in a given part of t he imag e: sediment (ro ck, cobble, sa nd, silt, ripple -either hor izontal, vertical or at 4 5 de- grees), sha dow or other features (t ypically ship- wrecks). All se diments ar e giv en with three cer- taint y levels (sure, mo dera tely sure or not sure), and the b ounda r y b etw een tw o sediments is als o given with a cer ta int y (sure, modera tely sur e or not sure). Hence, every pix e l of every image is lab eled as b eing either a certa in type of sediment or a shadow, or a bo undary with one of the three certaint y levels. Figure 1 g ives an ex ample of such a segmentation provided b y the exp ert. 4.2. Classi fication approac h The classification a ppr oach is ba sed on super- vised class ific a tion. In order to teach the class ifie r we hav e ra ndo mly divided the database in to tw o parts. On the learning database we hav e consid- ered, on r andomly chosen images only , the homo- geneous tiles w ith a 32 × 32 size a nd with a sure or mo der ately sure certitude level until to g et ap- proximately the same n umber of tiles in the lea rn- ing a nd test databases . On the test databa se we hav e consider ed tiles with a 32 × 32 s iz e and a re- cov ering step of 4. O n each tile we have extracted some features by a wav elet decomp osition. Ev aluation of Uncertain Imag e Cla ssification and Segmentation Algorithms 9 The discr ete translation inv ar ia nt wav elet transform is ba sed on the choice of the o ptimal translation for e ach decomp osition level. Each decomp osition le vel d g ives four new ima ges. W e choose here a deco mpo sition level d = 2. F o r ea ch image I i d (the i th image of the decomp osition d ) we ca lculate three features. The ener gy is giv en by: 1 N M N X n =1 M X m =1 I i d ( n, m ) , (12) where N and M are respe ctively the num ber of pixels on the rows, and on the co lumns. The en- tropy is estimated by: − 1 N M N X n =1 M X m =1 I i d ( n, m ) ln( I i d ( n, m )) , (13) and the mean is given by: 1 N M N X n =1 M X m =1 | I i d ( n, m ) | . (14) Consequently we obtain 15 features (3+4 *3). The ch osen class ifier is based on a Supp or t V ecto r Mac hine. The algorithm used her e is de- scrib ed in [28]. It is a one- vs -one m ulti-class ap- proach, and we take a linear kernel with a con- stant C = 1. W e hav e considered o nly three classes for learn- ing and tests: - cla ss 1: Rock a nd Cobble - cla ss 2: Ripple in all directions - cla ss 3: Sand and Silt Hence sha dow is not co nsidered and so the classi- fication can not b e go o d on tiles with shadow. In order to take in to accoun t unknown class es, one solution is to add a rejected class in the classifier. How ev er, as w e show farther do wn, we can also take into account this class if the classifier has no rejected class. The units of the cla ssifier are tiles with a 32 × 32 size with a rec ov ering step of 4. Hence, w e can classify tiles with a 4 × 4 size , considering the tile of 4 × 4 s ize in the middle on each tile of 32 × 32. 4.3. Ev aluation Figure 5 shows the result of the classification o f the s ame imag e than the one given in the figure 1. Sand (in red) and ro ck (in blue) a re q uite well classified but ripple (in yellow) is not well segmented. The dark blue corr esp onds to that part of the imag e that was not consider ed for the classification. Figure 5. Automatic seg men ted image. Just b y lo o king this figure 5 we c annot say whether the cla ssification is g o o d or not, and any decision stays very sub jectiv e. Mor eov er, the classification algor ithm could b e go o d for this im- age and not for o thers. So we pr op ose to use our measures. The used weigh ts here for the ce r titude are resp ectively 2/3 for sur e , 1/2 for mo derately sure and 1/3 for not sure. But other weights can be preferred acco rding to the a pplication. The nor malized confusion matrix obtained for one randomly par tition o f the database is given by:     40 . 51 5 . 77 53 . 72 19 . 65 18 . 79 6 1 . 56 3 . 51 1 . 15 95 . 34 45 . 96 12 . 47 4 1 . 57     (15) The last line means that ther e is s ha dow o r other parts clas sified in class 1, 2 or 3. W e can no te that a high pr o p o rtion of the ro ck or c obble (class 1) is classified a s sand or s ilt (class 3), a nd most o f the ripple (class 2) also. Sa nd and silt, the most com- mon kinds o f se diments on our images, are v ery 10 A. Martin et al. well classified. The vector of goo d classifica tion rate given by [4 0.51 18.79 9 5.34 0] and the vector of error clas sification rate given b y [41 .26 43 .8 4 28.47 5 0.00] summariz e these results. Where a s we have g o o d classifica tion for s and and silt, we also a lot of err o rs b ecause other sediments are classified as sand or silt. These r esults are not significan t enoug h in or- der to well ev aluate the obtained seg men tation. Our prop osed measures, g iven r e sp e c tiv ely b y the equations (6) and (11) expressed in pe r centage, are 65 .17 for the go o d detectio n criteria and 6 1.35 for the false ala rm criteria , if we co nsider the di- rection bas ed on the GVF the prop osed mea sures give 63.11 fo r go o d detection c r iteria and 64.84 for the false ala rm cr iteria. T o b etter illustrate these tw o last measure s, we hav e pro ceeded to four more randomly par titions. W e obta in a mean of 63.53 for the go o d detec- tion criteria with 3.37 for the standa rd deviation and a mean of 60.5 3 for the false a larm criteria with 7 .72 f or the s ta ndard dev iation. If w e con- sider the directio n based o n the GVF, we obtain a mean of 60 .09 for the go o d detection criteria with 3.1 3 for the standa rd deviation and a mean of 52.62 for the false alarm cr iteria with 8.04 for the standa rd deviation. The standard deviations show tha t the go o d detection criteria is more sta- ble than the false alarm criteria. Our t w o mea- sures c a n well ev aluate the go o d detection and the false a la rm. When we consider the direction based on the GVF, the criteria decrease b e cause of the weights given by the directio ns . Here, the deducted segmentation is dependent of the s ize of the tile, in th is case it could b e better to no t consider the directio n ba sed on the GVF. In o rder to ev aluate the class ifier appr oach, all these measures hav e to b e compar ed to the same measures calculated for other pa rameteriza tio ns or for other class ifier algorithms. 5. CONCLUSION W e ha ve prop osed so me new ev alua tion mea- sures for image class ifica tion and s e gmentation in uncertain en vironments. These new ev aluation measures can take into ac count the uncerta in la- bels . The prop osed classification e v aluation can be us ed for every kind o f uncertain elemen ts c las- sification and our segmentation ev alua tion can be used for all ima g e se gmentation approaches. W e hav e shown that a global image classification ev a l- uation must b e made by the e v alua tion of the classification and, a t the same time, by the ev a l- uation o f the pro duced segmen tation. The pro - po sed confusion matrix take in to account the un- certaint y of the exp ert and also the inhomoge- neous units ( e.g. tiles in the ca se o f lo cal image classification). Moreover we have defined go o d classification and erro r clas s ification rates from our confusion matrix . The prop osed segmenta- tion ev alua tion considers go o d a nd false detection bo undary measures where the s ub jectivity of the exp ert is co nsidered by the g iven uncertaint y o n the b oundaries. The fusion of the informa tion provided by v ar- ious expe r ts in our pr o p o sed ev alua tio n appr oach is made after a n individual ev aluation, which means that we fuse o ur differe nt measures cal- culated for eac h ex p er t. This fusion is made by using a simple sum: the uncertaint y is consid- ered directly in our measur es. W e can imag - ine fusing the informa tion provided by e x per ts befo re the ev a luation in order to obtain uncer- tain and/or imprecise reality ( e.g. defining fuzzy zones ar ound the b oundaries according to the cer- taint y given by th e expe rts). The fusion c a n be made a lso b y belief functions defined from the un- certainties. In this case w e ha v e to redefine our prop osed measures . F or instance, the r eality ob- tained b y the fusion of exp erts could b e used to outp e rform the learning step of the cla ssification. REFERENC ES 1. Y.J . Zha ng, A survey on ev aluation meth- o ds for ima ges seg men tation, Pattern Recog- nition, V ol. 29, No. 8 (1996 ), 133 5-134 6. 2. A. Martin, Co mparative study of information fusion metho ds for sonar images c la ssifica- tion, The Eighth In ternational Conference on Information F usion, Philadelphia, USA, 2 5-29 July 2005. 3. J .C. Russ, The Image Pr o cessing Handb o ok, CRC Press, 20 02. 4. H. Laanay a, A. Martin, D. Abo uta jdine, and Ev aluation of Uncertain Imag e Cla ssification and Segmentation Algorithms 11 A. K henchaf, A new dimensionality reduction metho d for se a b e d c harac ter ization: sup er- vised curvilinear comp onent ana ly sis, IE EE OCEANS’05 EUROPE, Br est, F rance, 20- 23 June 200 5. 5. A. Da vid and B. Lerner, Suppo rt v ector machine-based imag e classifica tio n for g enetic syndrome diagnosis, Pattern Recog nitio n Let- ters, V o l. 26, Issue 8 (200 5), 10 2 9-103 8. 6. M. Mavrofor akis, H. Georg iou, N. Dim- itrop oulos, D. Cav ouras and S. Theodo r - idis, Sig nifica nce analy s is of qualitative mam- mographic features, using linear classifier s , neural netw orks and suppo rt vector ma- chines, Europ ea n Journal of Ra diology , V ol. 54 (2005 ), 80-8 9. 7. S.B. P ark, J .W. Lee and S.K. Kim, conten t- Based image cla s sification using a neural net- work, Pattern Recognition Letters, V ol. 25 , Issue 3 (200 4), 28 7-300 . 8. Y.V. V enk atesh and S. Kumar Ra ja, O n the clas sification of mult isp ectral s atellite im- ages us ing the mult ilay er p erceptro n, Pattern Recognition, V ol. 36, No. 8 (2003), 2161-2 1 75. 9. I. Leblond, M. Leg r is and B. Sola iman, Use of classification and seg men tation of sidescan sonar ima ges for long ter m registration, IEEE OCEANS’05 EUROPE, Br est, F rance, 20- 23 June 200 5. 10. N.J . Pizzi, F uzzy pre-pr o cessing of go ld stan- drads as applied to biomedical clas sification, Artificial Intelligence in Medecine, V ol. 16 (1999), 171 -182. 11. T . Denœux, A Neur al Net w ork Classifier Based on Dempster -Shafer Theory , IEEE T r ansactions o n Systems, Man, a nd Cyb er- netics - P art A: Systems and Humans, V ol. 30, Issue 2 (2000 ), 1 31-15 0. 12. P . V anno or en b erghe, and Ph. Smets, Par- tially sup erv ised learning by a cre da l EM ap- proach, ECQSARU 2005, Bar c elona, Sp ain , july 2005 . 13. O . Alata and C. Ramananjarasoa, Unsuper- vised textured images seg men tation using 2-D quarter plane autor egressive mo del with four prediction supp orts, Pattern Recog nition Let- ters, V o l. 26, Issue 11 (20 05), 1 0 69-10 81. 14. J . F. Cann y , ”A computationa l approach t o edge detection,” IE EE T r ansactions on Pat- tern Analys is and Machine Intelligence, v ol. 8, 1986. 15. P . Kovesi, ”Image F eatures F rom Pha se Con- gruency”. Videre: A J ournal of Computer Vi- sion Research. MIT Press. V olume 1, Num b e r 3, Summer 199 9 . 16. J.B. Mena and J .A. Ma lpica, Color image seg - men tation based on three levels of texture sta- tistical ev a lua tion, Applied Mathematics and Computation, V o l. 161 (2005), 1- 1 7. 17. A. Bha le r ao a nd R. Wilson, Unsuper vised image segmentation co m bining reg ion and bo undary estimatio n, Image and Vision Com- puting, V o l. 19, Issue 6 (200 1), 35 3-368 . 18. H. Zang, J.E. F ritts and S.A. Goldman, A Co-ev aluation F r amework for Improving Seg- men tation Ev aluation, SPIE Defense and Se- curity Sympos ium - Signal Pro cessing Sensor F usio n, and T arget Recognition XIV (2005), 420-4 30. 19. Y.J. Zhang, Ev aluation and comparison of different segment ation alg orithms, Pattern Recognition Letters, V ol. 18, Issue 1 0 (1997), 963-9 74. 20. Y. Yitzhak y and E . Peli, A Metho d for Ob- jective Edge Detection Ev aluation and Detec- tor Parameter Selection, IE EE T ransa ctions on Pattern Analysis a nd Ma chine Intelligence, V ol. 25, No 8 (2003), 1027 -1033 . 21. R. Rom´ an-Rold´ an, J.F. G´ omez-Lop era , C. A tae-alla h, J. Mart ´ ınez-Aroza and P .L. Luque-Esca milla , A mea sure of quality for ev a luating metho ds of seg men tation and edge detection, Pattern Reco gnition, V ol. 34, Issue 5 (2001), 969 -980. 22. M. Bouet, C. Djeraba, A. Khenchaf, and H. Briand, Shape Pro cessing and Imag e Re- triev a l, Revue de l’Ob jet, s pec ial edition, Herm` es Science Publications, V ol. 6, Issue 2 (2000), 145-1 70. 23. T. Peli and D. Malah, A Study of Edge De- tection Algorithms, Computer Graphics and Image Pro ces sing, V ol. 20 (198 2), 1-2 1. 24. T. Kanoung o, M.Y. Jais imha, J. Palmer a nd R.M. Haralick, A Metho dology for Quantita- tive Performance E v alua tion o f Detection Al- gorithms, IEEE T r ansactions on Image P ro- 12 A. Martin et al. cessing, V ol. 4, N 21 (1995 ), 16 67-16 73. 25. C. X u and J.L. Prince, Snakes, Sha pe s , and Gradient V ector Flow, IEEE T r ansactions on Image Pro cessing , V ol. 7, Issue 3 (1998), 359- 369. 26. G. Le Chenadec, and J.M. Boucher, Sonar Image Segmentation using the Angular De- pendenc e o f Backscattering Distributions, IEEE OCEANS’05 E UR OPE, Bre st, F rance, 20-23 June 200 5 . 27. M. Lia nantonakis, and Y.R. Petillot, Sidescan sonar seg men tation using active contours and level set metho ds, IEEE O CEANS’05 EU- R OPE , Brest, F rance, 20- 23 J une 2 005. 28. C.C. Chang, and C.J. Lin, Lib- svm: a library for supp or t v ec- tor machines, Soft ware a v ailable at ht tp://www.csie.ntu.edu.t w/cjlin/libsvm, 2001.