Human expert fusion for image classification

HUMAN EXPER TS FUSION FOR IMA GE CLASSIFICA TION Arnaud MAR TIN and Christophe OSSW ALD Abstract In image classification, merging the opinion of se veral human experts is very important for different tasks such as the ev aluation or the training. Indeed, the ground truth is rarely kno wn before the scene imag ing. W e propose here differ- ent models i n order to fuse the informations giv en by two or more experts. The considered unit for the classification, a small tile of the image, can contain one or more kind of the considered classes giv en by the expe rts. A second problem that we hav e to take into account, is the amoun t of certainty of the expert has for each pixel of the tile. In order to so lve these problems we de fine fiv e mod els in the con - text of the Dempster-Shafer Theory and in the context of the Dezert-Smarandache Theory and we study the possible decisions with these models. K eywords: E xperts fusion, DST , DSmT , image classification. Introd uction Fusing the opinion o f several human experts, also kn own as the experts fusion problem , is an important question in the image classification field and very fe w studied. In deed, the g roun d truth is rarely known befo re the scene has b een ima ged; conseque ntly , some experts have to provide their per ception of the images in ord er to train the classifiers (for super vised classifiers), and also to ev aluate the image classification. In most of the real applicatio ns, the experts cannot provide th e different classes o n the images with cer titude. Mo reover , the difference of experts percep tions can b e very large, and so many parts of the imag es have conflictin g inf ormation . Thereby , only on e expert r eality is not reliable enough , and experts fusion is required. Image classification is ge nerally done on a lo cal part of the image (pixel, o r most of the tim e on small tiles of e.g. 16 × 16 or 32 × 32 pixels). Classification m ethods can usually be described into three steps. First, s ignifican t fe atures are extracted from these tiles. Generally , a secon d step in nece ssary in ord er to redu ce th ese featu res, because they are too numer ous. In the third step, these f eatures are given to classification algorithm s. The particularity in considerin g small tiles in im age classification is that sometimes, more than one class can co-exist on a tile. An example of such an im age classification pro cess is seabed char acterization. This serves many usef ul p urposes, e.g h elp the navigation of Autonom ous Underwater V e- hicles or provid e data to sedimen tologists. In such sonar applications, which serve as examples thr ough out the p aper, seabed images are o btained with many im perfection s INFORMA TION & SECURITY . An International J ournal, V ol. 20 , 2006, 1-22 2 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION [1]. In deed, in order to build images, a hu ge number of physical data (geometry of the device, co ordina tes of the ship, m ovements of the sonar, etc. ) are taken into acco unt, but these data ar e p olluted with a large amou nt o f n oises caused by in strumentatio ns. In addition, there are some interfer ences due to the signal traveling on multip le path s (reflection on the bottom or surface), d ue to speckle, and du e to f auna and flora. Th ere- fore, son ar images ha ve a lot of impe rfections su ch as imprecision an d uncertainty ; thus sed iment classification o n son ar imag es is a difficult problem. In this kind of applications, the re ality is un known and different exper ts can p ropose different clas- sifications of the imag e. Figur e 1 exhibits th e differences between the in terpretatio n and the certainty of two sonar experts tryin g to differentiate the type of sediment (ro ck, cobbles, san d, ripp le, silt) or shadow wh en the inf ormation is invisible. Each color correspo nds to a k ind of sediment and the associated certainty of the expert for this sediment expressed in term of sur e, moderately sure and not sure. Thus, in order to learn an automatic classification a lgorithm, we mu st take into account this difference and the un certainty of each expert. For example, how a tile of rock labeled as not sure must be taken into acc ount in the learn ing step o f the classifier a nd h ow to take into account this tile if another expert s ays that it is sand? An other problem is: how to take into account the tiles with more than one sediment? Figure 1: Segmen tation gi ven by tw o experts. Many fu sion theo ries can be used for th e experts fusion in image classification such as voting rules [2, 3], possibility theo ry [4, 5], belief function th eory [6, 7]. In our case, experts can express the ir cer titude on their p erception . As a result, pro babilities theories such as the Bayesian theory o r the belief function theory are more a dapted. Indeed , the possibility theo ry is mo re ad apted to imitate the imprecise data whereas probab ility-based theories is more adap ted to imitate the u ncertain data. Of course both possibility and prob ability-based theories can imitate imprecise and unc ertain data Arnaud MARTIN and Christophe OSSW ALD 3 at the same time, but not so easily . That is why , our cho ice is con ducted on the b elief function theor y , also called th e Dempster-Shafer theory (DST) [6, 7]. W e can di vide the fusion appro ach into f our steps: the belief functio n m odel, the pa rameters estimation depend ing on the model (not always n ecessary), th e combination , and the decision. The most difficult step is pr esumably the first on e: the b elief fun ction model f rom which the other steps follow . Moreover , in real ap plications of ima ge classification, expe rts conflict can be very large, and we ha ve to take into account th e heterog eneity o f the tiles (more than one class can b e presen t on the tile). Consequently , the Dez ert-Smaran dache Th eory (DSmT) [8], an extension of the belief function theory , can fit better to our problem of image classification if ther e is conflict. I ndeed, consid ering the space of discern ment Θ = { C 1 , C 2 , . . . , C n } , where C i is the hypothesis “the considered unit b elongs to the class i ” . I n the classical be lief function theo ry , the belief fun ctions, also called th e basic belief assignments, are defined by a m apping of the power set 2 Θ onto [0 , 1] . The power set 2 Θ is closed under the ∪ operato r , and ∅ ∈ 2 Θ . In the extension pro posed in the DSmT , g eneralized basic belief assignm ents ar e defined by a map ping of the hyper-power set D Θ onto [0 , 1 ] , where t he hyp er-power set D Θ is closed under both ∪ and ∩ operato rs. Conseque ntly , we can man age fin ely the c onflict o f the exper ts a nd also take into account the tiles with more than one class. In the fir st section, we discuss and presen t different belief function mod els based on th e p ower set an d th e h yper p ower set. T hese mo dels try to an swer ou r pro blem. W e study these mo dels also in the steps of comb ination and decision of the informatio n fusion. These m odels allow , in a seco nd section , to a gen eral discussion on th e differ- ence between the DSmT and DST in terms of capacity to re present our problem and in terms of decision. Finally , we present an illustration of our propo sed experts fusion on real sonar images, which represent a particularly uncertain en viro nment. 1 Our prop osed Models In this section, we presen t five mod els taking in to account the possible specificities of the application. First, we r ecall the princip les of the DST and DSmT we a pply here. The n we present a nu merical e xamp le which illustrates the fi ve prop osed models presented afterward. Th e first three models are pr esented in the context o f the DST , the fourth model in the context of the DSmT , and the fifth model in both contexts. 4 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION Theory Bases Belief Function Models The belief f unction s or basic belief a ssignments m are defined by the mappin g of the power set 2 Θ onto [0 , 1] , in the DST , an d by th e mappin g of the hyp er-power set D Θ onto [0 , 1 ] , in the DSmT , with : m ( ∅ ) = 0 , (1) and X X ∈ 2 Θ m ( X ) = 1 , (2) in the DST , and X X ∈ D Θ m ( X ) = 1 , (3) in the DSmT , where X is a given tile of the image. The eq uation (1) allows that we a ssume a closed world [7, 8 ]. W e can define the belief function with only: m ( ∅ ) > 0 , (4) and the world is op en [9]. In a closed world, we can add one element in order to propo se an open world. These simp le cond itions in equ ation (1) and ( 2) or (1 ) and (3), giv e a large pan el of definitions of the belief fun ctions, which is one of the difficulties of the theory . Th e belief function s must theref ore be chosen according to the intended application. In our case, the spac e of discernmen t Θ repr esents the different kind of sediments on sonar images, such as ro ck, sand, silt, cobble, ripple or shad ow (that means no sediment inf ormation ). Th e experts g iv e their perception and b elief accor ding to their certainty . For instance, the expert can be moder ately sure of his choice when he labels one part of the image as belonging to a certain class, and be totally doubtful on another part of the image. Moreover , on a consid ered tile, more than o ne sediment can be present. Consequently we have to take into account all these aspects of th e application s. In order to simplify , we consider only two classes in the fo llowing: the rock referred as A , and the sand , referred as B . Th e proposed models can be easily extended , but their study is easier to understand with only two classes. Hence, on certain tiles, A and B c an b e pr esent for one o r more experts. The belief fun ctions have to take into accou nt th e certain ty given b y the experts (referre d respectively as c A and c B , two nu mbers in [0 , 1] ) as well as the pr oportio n of the kind of sediment in the tile X ( referred as p A and p B , also two numbers in [0 , 1] ). W e have Arnaud MARTIN and Christophe OSSW ALD 5 two in terpretation s of “the expert believes A ”: it can mean that the expe rt thin ks that there is A on X an d no t B , o r it can mean that the expert thinks that th ere is A on X an d it can also have B but he does not say anything about it. T he first interpretatio n yields that hypothe ses A and B are exclusive an d with the second they are not exclusi ve. W e only study the first case: A and B ar e exclusiv e. But on the tile X , the exp ert can also provide A an d B , in this case the two pro positions “the expert believes A ” a nd “the expert belie ves A and B ” are not exclusive. Combination rules Many comb ination rules h ave been pro posed these last y ears in the context of the belief function theory ( [10, 11, 9, 12, 8, 13], etc. ). In the con text of the DST , the combinatio n rule m ost u sed tod ay seem s to be th e con junctive consen sus rule given b y [9] fo r all X ∈ 2 Θ by: m ( X ) = X Y 1 ∩ ... ∩ Y M = X M Y j =1 m j ( Y j ) , (5) where Y j ∈ 2 Θ is the re sponse of the expert j , and m j ( Y j ) the associated b elief fun c- tion. In the co ntext of the DSmT , the conjun ctiv e consensus rule ca n be used for all X ∈ D Θ and Y ∈ D Θ . If we want to take the decision o nly on th e elements in Θ , some rules pro pose to r edistribute the con flict on these e lements. T he mo st acco mplished rule to provide tha t is the PCR5 gi ven in [13] for two e xperts and for X ∈ D Θ , X 6 = ∅ by: m P C R 5 ( X ) = m 12 ( X )+ X Y ∈ D Θ , c ( X ∩ Y )= ∅  m 1 ( X ) 2 m 2 ( Y ) m 1 ( X ) + m 2 ( Y ) + m 2 ( X ) 2 m 1 ( Y ) m 2 ( X ) + m 1 ( Y )  , (6) where m 12 ( . ) is the con junctive consensus rule giv en by the eq uation (5), c ( X ∩ Y ) is the conjunctive nor mal form of X ∩ Y and the denominato rs ar e not null. W e can easily genera lize this rule for M experts, for X ∈ D Θ , X 6 = ∅ : m P C R 6 ( X ) = m ( X ) + (7) M X i =1 m i ( X ) 2 X M − 1 ∩ k =1 Y σ i ( k ) ∩ X ≡∅ ( Y σ i (1) ,...,Y σ i ( M − 1) ) ∈ ( D Θ ) M − 1        M − 1 Y j =1 m σ i ( j ) ( Y σ i ( j ) ) m i ( X ) + M − 1 X j =1 m σ i ( j ) ( Y σ i ( j ) )        , 6 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION where σ i counts from 1 to M av oiding i :  σ i ( j ) = j if j < i, σ i ( j ) = j + 1 if j ≥ i, (8) m i ( X ) + M − 1 X j =1 m σ i ( j ) ( Y σ i ( j ) ) 6 = 0 , a nd m is the co njunctive consensus rule given by the equation (5). The com parison of all the comb ination rules is not th e purp ose of this paper . Co n- sequently , we use he re the eq uation (5) in th e con text of the DST and th e equ ation (7) in the context of the DSmT . Decision rules The decision is a difficult task. No measures are able to provide the b est decision in all the cases. Generally , we con sider the maximum of one of th e th ree functio ns: credibility , plausibility , and pignistic prob ability . In the context of the DST , the credibility function is gi ven for all X ∈ 2 Θ by: bel( X ) = X Y ∈ 2 X ,Y 6 = ∅ m ( Y ) . (9) The plausibility function is given f or all X ∈ 2 Θ by: pl( X ) = X Y ∈ 2 Θ ,Y ∩ X 6 = ∅ m ( Y ) = bel (Θ ) − b el ( X c ) , (10) where X c is the complem entary of X . Th e pignistic probab ility , introd uced b y [14], is here given fo r all X ∈ 2 Θ , with X 6 = ∅ by: betP ( X ) = X Y ∈ 2 Θ ,Y 6 = ∅ | X ∩ Y | | Y | m ( Y ) 1 − m ( ∅ ) . (11) Generally th e max imum o f th ese fu nctions is taken on the ele ments in Θ , but we will giv e the values on all the focal elements. In th e context of the DSmT the correspond ing generalized fun ctions have been propo sed [15, 8]. The generalized credibility B el is defined by: Bel( X ) = X Y ∈ D X m ( Y ) (12) The generalized plausibility P l is defined by: Pl( X ) = X Y ∈ D Θ ,X ∩ Y 6 = ∅ m ( Y ) (13) Arnaud MARTIN and Christophe OSSW ALD 7 The gen eralized p ignistic prob ability is given for all X ∈ D Θ , with X 6 = ∅ is d efined by: GPT( X ) = X Y ∈ D Θ ,Y 6 = ∅ C M ( X ∩ Y ) C M ( Y ) m ( Y ) , (14) where C M ( X ) is the DSm cardina lity cor respond ing to th e number of parts of X in the V enn diagra m of the pr oblem [15, 8]. If the credibility functio n pr ovides a pessimist d ecision, the plausibility function is ofte n too optimist. The pignistic p robability is o ften taken as a compromise. W e present the three functions for our models. Numerical and illustrative e xam ple Consider two exper ts provid ing their opin ion on the tile X . The first expert say s that on tile X ther e is some rock A with a certainty equal to 0.6. Hence for this first expert we h ave : p A = 1 , p B = 0 , and c A = 0 . 6 . The secon d expert thin ks that there a re 50% of rock an d 5 0% of sand on the c onsidered tile X with a r espective certainty of 0.6 and 0.4. Hen ce for the secon d expert we ha ve: p A = 0 . 5 , p B = 0 . 5 , c A = 0 . 6 and c B = 0 . 4 . W e illustrate all our propo sed mode ls with this numerical ex emple. Model M 1 If we con sider the sp ace of d iscernment given by Θ = { A, B } , w e can defin e a b elief function by: if the expert says A :  m ( A ) = c A , m ( A ∪ B ) = 1 − c A , if the expert says B :  m ( B ) = c B , m ( A ∪ B ) = 1 − c B . (15) In this case, it is natural to distribute 1 − c A and 1 − c B on A ∪ B which represent the ignoran ce. This m odel takes into accou nt the certainty g iv en by the expert but th e sp ace o f discernmen t do es no t con sider th e p ossible h eterogen eity of the given tile X . Con se- quently , we have to ad d a nother focal elem ent m eaning that th ere are two classes A and B on X . In the con text of th e Demp ster-Shafer theory , we can c all this focal eleme nt C an d the space of discernment is gi ven by Θ = { A, B , C } , and the power set is gi ven 8 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION by 2 Θ = {∅ , A, B , A ∪ B , C , A ∪ C, B ∪ C, A ∪ B ∪ C } . Hence we can define our first model M 1 for our application by: if the expert s ays A :  m ( A ) = c A , m ( A ∪ B ∪ C ) = 1 − c A , if the expert s ays B :  m ( B ) = c B , m ( A ∪ B ∪ C ) = 1 − c B , if the expert s ays C :  m ( C ) = p A .c A + p B .c B , m ( A ∪ B ∪ C ) = 1 − ( p A .c A + p B .c B ) . (16) On our numer ical e xamp le, we obtain : A B C A ∪ B ∪ C m 1 0 . 6 0 0 0 . 4 m 2 0 0 0 . 5 0 . 5 Hence for the consensus combin ation fo r th e model M 1 , the be lief fu nction m 12 , the credibility , the plausib ility and the pignistic probability are gi ven by: el ement m 12 bel pl b etP ∅ 0 . 3 0 0 − A 0 . 3 0 . 3 0 . 5 0 . 5238 B 0 0 0 . 2 0 . 0 952 A ∪ B 0 0 . 3 0 . 5 0 . 6 190 C 0 . 2 0 . 2 0 . 4 0 . 3810 A ∪ C 0 0 . 5 0 . 7 0 . 9 0 48 B ∪ C 0 0 . 2 0 . 4 0 . 4 762 A ∪ B ∪ C 0 . 2 0 . 7 0 . 7 1 Where: m 12 ( ∅ ) = m 12 ( A ∩ C ) = 0 . 30 . (17) This belief function provides an ambig uity be cause the sam e mass is put o n A , the rock, an d ∅ , the conflict. Wit h the maximu m of credibility , plausibility or pignistic probab ility this am biguity is suppressed becau se th ese functio ns do not c onsider the empty set. Arnaud MARTIN and Christophe OSSW ALD 9 Model M 2 In the first mod el M 1 , the po ssible heterogeneity of the tile is taken into acco unt. Howe ver , th e ignora nce is character ized by A ∪ B ∪ C an d not by A ∪ B anymo re, an d class C rep resents th e situation when the two c lasses A and B are on X . Con sequently A ∪ B ∪ C c ould be equal to A ∪ B , and we can prop ose another model M 2 giv en by: if the expert s ays A :  m ( A ) = c A , m ( A ∪ B ) = 1 − c A , if the expert s ays B :  m ( B ) = c B , m ( A ∪ B ) = 1 − c B , if the expert s ays C :  m ( C ) = p A .c A + p B .c B , m ( A ∪ B ) = 1 − ( p A .c A + p B .c B ) . (18) On our numerical example, we have: A B C A ∪ B m 1 0 . 6 0 0 0 . 4 m 2 0 0 0 . 5 0 . 5 In this mo del M 2 the ign orance is par tial and th e conjunc ti ve co nsensus rule, the credibility , the plausib ility and the pignistic probability are gi ven by: el ement m 12 bel pl betP ∅ 0 . 5 0 0 − A 0 . 3 0 . 3 0 . 3 0 . 6 B 0 . 2 0 . 2 0 . 2 0 . 4 A ∪ B 0 0 . 5 0 . 5 1 C 0 0 0 0 A ∪ C 0 0 . 3 0 . 3 0 . 6 B ∪ C 0 0 . 2 0 . 2 0 . 4 A ∪ B ∪ C 0 0 . 5 0 . 5 1 where m 12 ( ∅ ) = m 12 ( A ∩ C ) + m 12 ( C ∩ ( A ∪ B )) = 0 . 30 + 0 . 2 = 0 . 5 . (19) The previous a mbiguity in M 1 between A (the rock) and ∅ (the con flict) is still present with a be lief on ∅ highe r than A . Mo reover , in this m odel the mass o n C is null! 10 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION These models M 1 and M 2 are dif feren t because in th e DST the classes A , B and C are supposed to b e e xclusive. I ndeed, the f act that the power set 2 Θ is not closed under ∩ operator leads to the e xclusivity of the classes. Model M 3 In our application, A , B and C cannot b e conside red exclusiv e on X . In or der to propo se a mo del f ollowing the DST , we have to study exclusive classes only . Henc e, in ou r application , we can co nsider a space of discernmen t of th ree exclusive classes Θ = { A ∩ B c , B ∩ A c , A ∩ B } = { A ′ , B ′ , C ′ } , following the no tations giv en o n the figure 2. Figure 2: No tation of the intersection of two classes A and B . Hence, we can pro pose a ne w model M 3 giv en by: if the expert s ays A :  m ( A ′ ∪ C ′ ) = c A , m ( A ′ ∪ B ′ ∪ C ′ ) = 1 − c A , if the expert s ays B :  m ( B ′ ∪ C ′ ) = c B , m ( A ′ ∪ B ′ ∪ C ′ ) = 1 − c B , if the expert s ays C :  m ( C ′ ) = p A .c A + p B .c B , m ( A ′ ∪ B ′ ∪ C ′ ) = 1 − ( p A .c A + p B .c B ) . (20) Arnaud MARTIN and Christophe OSSW ALD 11 Note that A ′ ∪ B ′ ∪ C ′ = A ∪ B . On ou r numerical example we obtain: A ′ ∪ C ′ B ′ ∪ C ′ C ′ A ′ ∪ B ′ ∪ C ′ m 1 0 . 6 0 0 0 . 4 m 2 0 0 0 . 5 0 . 5 Hence, the con junctive consensus rule, th e credibility , the plausibility and the pig- nistic prob ability are gi ven by: el ement m 12 bel pl b etP ∅ 0 0 0 − A ′ = A ∩ B c 0 0 0 . 5 0 . 2 167 B ′ = B ∩ A c 0 0 0 . 2 0 . 0 667 A ′ ∪ B ′ = ( A ∩ B c ) ∪ ( B ∩ A c ) 0 0 0 . 5 0 . 2833 C ′ = A ∩ B 0 . 5 0 . 5 1 0 . 7167 A ′ ∪ C ′ = A 0 . 3 0 . 8 1 0 . 9333 B ′ ∪ C ′ = B 0 0 . 5 1 0 . 7 833 A ′ ∪ B ′ ∪ C ′ = A ∪ B 0 . 2 1 1 1 where m 12 ( C ′ ) = m 12 ( A ∩ B ) = 0 . 2 + 0 . 3 = 0 . 5 . (21) On this example, with this model M 3 the decision will be A with the maximu m of pignistic prob ability . But the decision cou ld a priori b e taken also o n C ′ = A ∩ B because m 12 ( C ′ ) is the h ighest. W e show h owe ver in the discussion section th at it is not possible. Model M 4 In the con text of the DSmT , we can write C = A ∩ B an d easily propo se a fourth model M 4 , without any consideration on the exclusi vity of the class es, given by: if the expert s ays A :  m ( A ) = c A , m ( A ∪ B ) = 1 − c A , if the expert s ays B :  m ( B ) = c B , m ( A ∪ B ) = 1 − c B , if the expert s ays A ∩ B :  m ( A ∩ B ) = p A .c A + p B .c B , m ( A ∪ B ) = 1 − ( p A .c A + p B .c B ) . (22) 12 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION This last model M 4 allows to represent o ur pro blem witho ut ad ding a n ar tificial class C . Thu s, the model M 4 based on the DSmT gives : A B A ∩ B A ∪ B m 1 0 . 6 0 0 0 . 4 m 2 0 0 0 . 5 0 . 5 The obtained mass m 12 with the conjunctive co nsensus yields: m 12 ( A ) = 0 . 30 , m 12 ( B ) = 0 , m 12 ( A ∩ B ) = m 1 ( A ) m 2 ( A ∩ B ) + m 1 ( A ∪ B ) m 2 ( A ∩ B ) = 0 . 30 + 0 . 20 = 0 . 5 , m 12 ( A ∪ B ) = 0 . 2 0 . (23) These results a re e xactly th e same for the model M 3 . These two mod els do not present ambiguity and show that the mass on A ∩ B (rock and sand) is the highest. The generalized credib ility , the generalized p lausibility and th e generalized pignis- tic proba bility are given by: el ement m 12 Bel Pl GPT ∅ 0 0 0 − A 0 . 3 0 . 8 1 0 . 933 3 B 0 0 . 5 0 . 7 0 . 7833 A ∩ B 0 . 5 0 . 5 1 0 . 7 167 A ∪ B 0 . 2 1 1 1 Like the model M 3 , on th is examp le, th e decision will be A with the maxim um of pign istic proba bility criteria. But here also the max imum of m 12 is reach ed for A ∩ B = C ′ . If we want to c onsider o nly the kind of p ossible sedimen ts A and B and no t also t he conjunc tions, we can use a propo rtional conflict redistribution rules such as th e PCR5 propo sed in [13]. Conseque ntly we have x = 0 . 3 . (0 . 5 / 0 . 3) = 0 . 5 and y = 0 , and the PCR5 rule provides: m P C R 5 ( A ) = 0 . 30 + 0 . 5 = 0 . 8 , m P C R 5 ( B ) = 0 , m P C R 5 ( A ∪ B ) = 0 . 2 0 . (24) Arnaud MARTIN and Christophe OSSW ALD 13 The credibility , the plausibility and the pignistic probab ility are given by: el ement m P C R 5 bel pl betP ∅ 0 0 0 − A 0 . 8 0 . 8 1 0 . 9 B 0 0 0 . 2 0 . 1 A ∪ B 0 . 2 1 1 1 On this numerical example, the decision will be th e same than the consensus rule, here the maximu m o f pignistic proba bility is reached for A (r ock). I n the next sectio n we see that is not always the case. Model M 5 Another model M 5 which can be used in both the DST and the DSmT is gi ven consid- ering only one belief function accord ing to the prop ortion by:    m ( A ) = p A .c A , m ( B ) = p B .c B , m ( A ∪ B ) = 1 − ( p A .c A + p B .c B ) . (25) If for one expert, the tile co ntains on ly A , p A = 1 , an d m ( B ) = 0 . If for another expert, the tile conta ins A and B , we ta ke into accou nt the certainty and p ropor tion of the two sediments but not only on one focal element. Con sequently , we have simply: A B A ∪ B m 1 0 . 6 0 0 . 4 m 2 0 . 3 0 . 2 0 . 5 In th e DST context, the con sensus ru le, the credibility , the plausibility and the pignistic proba bility are given by: el ement m 12 bel pl b etP ∅ 0 . 12 0 0 − A 0 . 6 0 . 6 0 . 8 0 . 7955 B 0 . 08 0 . 08 0 . 28 0 . 2045 A ∪ B 0 . 2 0 . 88 0 . 88 1 In this case we do no t hav e the plausibility to decide on A ∩ B , becau se the conflict is on ∅ . 14 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION In the D SmT context, the con sensus r ule, th e generalized cred ibility , the gen eral- ized plausibility and the generalized pignistic probability are given by: el ement m 12 Bel Pl GPT ∅ 0 0 0 − A 0 . 6 0 . 72 0 . 9 2 0 . 8 9 33 B 0 . 08 0 . 2 0 . 4 0 . 6333 A ∩ B 0 . 12 0 . 12 1 0 . 5267 A ∪ B 0 . 2 1 1 1 The decision with the maximum of pignistic probab ility criteria is still A . The PCR5 rule provides: el ement m P C R 5 bel pl b etP ∅ 0 0 0 − A 0 . 69 0 . 69 0 . 89 0 . 7 9 B 0 . 11 0 . 11 0 . 31 0 . 21 A ∪ B 0 . 2 1 1 1 where m P C R 5 ( A ) = 0 . 60 + 0 . 09 = 0 . 69 , m P C R 5 ( B ) = 0 . 08 + 0 . 03 = 0 . 11 . W ith this mode l and examp le the PCR5 ru le, the d ecision will be also A , and we do not have d ifference between the consen sus rules in the DST and DSmT . 2 Discussion W e h av e build, in the p revious section, the mo dels M 1 , M 2 , M 3 , M 4 , and M 5 in the DSmT ca se in or der to take into a ccount th e d ecision c onsidering also A ∩ B (“th ere is ro ck and sand on th e tile”). I n fact o nly the M 1 and M 2 models can do it. Mo del M 2 can do it only if both experts say A ∩ B . These two m odels assume that A , B and A ∩ B ar e exclusive. Of cou rse this assum ption is false. For th e mo dels M 3 , M 4 and M 5 , we have to take th e decision on the credibilities, plausibilities or pignistic probab ilities, but these th ree fu nctions for A ∩ B cannot be higher th an A or B (or for C ′ than A ′ ∪ C ′ and B ′ ∪ C ′ with the notatio ns o f the mod el M 3 ). Ind eed for all x ∈ A ∩ B , x ∈ A and x ∈ B , so f or all X ⊆ Y : bel( X ) ≤ bel( Y ) , pl( X ) ≤ pl( Y ) , betP ( X ) ≤ b etP( Y ) , Bel( X ) ≤ B el( Y ) , Pl( X ) ≤ Pl( Y ) , GPT( X ) ≤ GPT( Y ) . Arnaud MARTIN and Christophe OSSW ALD 15 Hence, ou r first p roblem is no t solved: we can never choose A ∩ B with the max- imum of cr edibility , p lausibility or p ignistic prob ability . If the two exp erts think that the considered tile contain s rock an d sand ( A ∩ B ), then the pignistic prob abilities are equal. Howev er the belief on A ∩ B can be the highest (see th e e xamp le on the models M 3 and M 4 ). Th e limits of the decision rules are reached in this case. W e have seen that we can describe ou r problem both in the DST and th e DSmT context. T he DSmT is more adapted to modelize the belief on A ∩ B fo r example with the mod el M 4 , but mode l M 3 with the DST ca n provide exactly the same belief on A , B and A ∩ B . Consequen tly , the only difference we can e xpect on the decision comes from the co mbination rules. In the presen ted numerical example, the decision s are the same: we ch oose A . An example of decision inst ability T ake ano ther example with this last model M 5 : The fir st expert provides: p A = 0 . 5 , p B = 0 . 5 , c A = 0 . 6 and c B = 0 . 4 , an d th e secon d expert p rovides: p A = 0 . 5 , p B = 0 . 5 , c A = 0 . 86 and c B = 1 . W e want take a decision only on A o r B . Hence we have: A B A ∪ B m 1 0 . 3 0 . 2 0 . 5 m 2 0 . 43 0 . 5 0 . 07 For M 5 on the DST context: el ement m 12 bel pl b etP ∅ 0 . 23 6 0 0 − A 0 . 3 65 0 . 365 0 . 4 0 . 5007 B 0 . 364 0 . 3 64 0 . 399 0 . 4993 A ∪ B 0 . 03 5 0 . 764 0 . 7 64 1 M 5 with PCR5 gives (with the partial co nflicts: x 1 = 0 . 0 562 , y 1 = 0 . 0937 , x 2 = 0 . 0587 and y 2 = 0 . 0937 ) : el ement m P C R 5 bel pl b etP ∅ 0 0 0 − A 0 . 4 79948 0 . 479 0 . 5 1 49 0 . 49 7 4 B 0 . 4850 52 0 . 485 0 . 5202 0 . 5 026 A ∪ B 0 . 035 1 1 1 This last e xam ple shows th at we ha ve a dif ference be tween the DST and t he DS mT , b ut what is the best solutio n? W ith the DST we cho ose A and with the DSmT we ch oose B . W e can show that the decision will be the same in the most o f the case (abo ut 99.4%). 16 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION Stability of decision pr ocess The space wh ere experts can define th eir o pinions on wh ich n classes ar e pr esent in a giv en tile is a part o f [0 , 1] n : E = [0 , 1] n ∩ ( X X ∈ Θ m ( X ) ≤ 1) . In order to stud y th e different co mbination rules, and th e situations wh ere the y differ , we use a Monte Carlo method, considering the weig hts p A , c A , p B , c B , . . . , as uniform variables, filtering them by the condition X X ∈ Θ p X c X ≤ 1 for one expert. Thus, we measure the pro portion o f situations where decision differs b etween the con sensus combinatio n ru le, and the PCR5, wh ere conflict is propor tionally dis- tributed. W e can not choose A ∩ B , as the measure of A ∩ B is always lower (or equal with probab ility 0) than the measure of A or B . I n the case of two classes, A ∪ B is th e ignoran ce, and is usually excluded ( as it alw ays maximises b el , pl , b etP , Bel , P l and GPT ). W e restrict the possible choices to singletons, A , B , etc. Th erefor e, it is equiv- alent to tag the tile by the most c redible class (maximal fo r b el ), the most plausible (maximal f or pl ), the mo st pro bable (max imal fo r b etP ) or the heaviest (max imal for m ), as the only focal elements are singletons, Θ and ∅ . The only situation where the total ord er indu ced by the m asses m o n singletons can be mo dified is when the con flict is distributed on the singleton s, a s is the case in the PCR5 method. Thus, for two classes, the subspace where the decision is “rock” by consensu s rule is very similar to the subsp ace where the d ecision is “r ock” by the PCR5 rule: only 0.6% o f the volume differ . For a higher n umber o f classes, the d ecision o btained by fusing the two experts’ opinions is much less stable: number of classes 2 3 4 5 6 7 decision change 0.6% 5.5% 9.1% 1 2.1% 14.6% 16 .4% Therefo re, the specificity of PCR5 appears mostly with mo re than two classes, and the different combination rules are nearly equiv alent when decision mu st be taken within two possible classes. Left part of figure 3 sho ws the density of conflict within E , for a number of classes of 2, 3 , 6 and 7. Right par t shows how this distribution ch anges if we restrict E to the cases wh ere the d ecision ch anges b etween c onsensus (dotted lines) an d PCR5 (p lain lines). Conflict is mo re importan t in this subspa ce, mostly b ecause a low conflict u su- ally m eans a clear decision : the m easure on the be st cla ss is often very different than measure on the second best class. For the “two exper ts and two classes” case, it is difficult to ch aracterize an alytically the stability of the decision pr ocess. Howe ver , we can easily show th at if m 1 ( A ) = Arnaud MARTIN and Christophe OSSW ALD 17 Figure 3: Density of co nflict for (left) un iform ran dom exper ts an d (right) d ata with different decision between consensus and PCR5. m 2 ( B ) or if m 1 ( A ) = m 1 ( B ) , the final decision does no t depend on the cho sen combinatio n rule. 3 Illustration Database Our database contains 40 so nar images provided by the GESMA (Group e d’Etude s Sou s-Marines de l’Atlantique) . These imag es were obtained with a Klein 5400 lateral sonar with a reso lution of 20 to 30 cm in a zimuth and 3 cm in ran ge. The sea-bottom depth was between 15 m and 40 m. T wo exper ts have manu ally segmented th ese images giving the kind of sedimen t (rock, cobble, sand, silt, ripple (horizontal, vertical o r at 45 degrees)), s hadow or other (typically ships) parts on images, helped by the manual segmentation in terface p re- sented in fig ure 4 . All s edimen ts are gi ven with a c ertainty level (sure, moderately sure or not sure). Hen ce, e very pixel of ev ery i mag e is labeled as being either a certain type of sediment or a shadow or other . Results W e n ote A = r ock, B = cobble, C = san d, D = silt, E = ripp le, F = shadow an d G = other, hen ce we ha ve se ven classes an d Θ = { A, B , C, D , E , F, G } . W e h ave 18 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION Figure 4: Man ual Segmentation Interface. applied the generalized model M 5 on tiles of size 32 × 32 giv en by:                        m ( A ) = p A 1 .c 1 + p A 2 .c 2 + p A 3 .c 3 , for rock , m ( B ) = p B 1 .c 1 + p B 2 .c 2 + p B 3 .c 3 , for cob ble, m ( C ) = p C 1 .c 1 + p C 2 .c 2 + p C 3 .c 3 , fo r ripple, m ( D ) = p D 1 .c 1 + p D 2 .c 2 + p D 3 .c 3 , fo r sand, m ( E ) = p E 1 .c 1 + p E 2 .c 2 + p E 3 .c 3 , fo r silt, m ( F ) = p F 1 .c 1 + p F 2 .c 2 + p F 3 .c 3 , for shadow , m ( G ) = p G 1 .c 1 + p G 2 .c 2 + p G 3 .c 3 , fo r other , m (Θ) = 1 − ( m ( A ) + m ( B ) + m ( C ) + m ( D ) + m ( E ) + m ( F ) + m ( G )) , (26) where c 1 , c 2 and c 3 are the weigh ts associated to the certitude r espectively: “sure”, “modera tely su re” and “not sure”. T he chosen weig hts are here: c 1 = 2 / 3 , c 2 = 1 / 2 an d c 3 = 1 / 3 . I ndeed we have to consider th e cases wh en the same kind o f sediment (but with dif feren t certainties) is present o n the sam e tile. The prop ortion of each sediment in the tile associated to these weights is n oted, for instance f or A : p A 1 , p A 2 and p A 3 . The table 1 gives th e con flict matrix o f the two exp erts. W e n ote that the mo st of con flict come fr om a difference of opinio n between sand an d silt. F or instance, the exper t 1 provides many tiles of sand w hen the expert 2 think s that is silt (conflict induced of 0.052 4). This conflict is explained by the diffi culty for the experts to differentiate sand and silt that differ with only the in tensity . Part of con flict com es also from the f act that ripples are hard to distinguish from sand or silt. Ripp les, that is, sand o r silt in a special c onfigur ation, is some times difficult to see on the image s, and the ripples are m ost of the time visible in a g lobal zone where sand o r silt is p resent. Cobbles also yield conflicts, especially wit h sand, silt and rock: cobble is described by some small ro cks on sand or silt. The total conflict between the two experts is 0.120 9. Arnaud MARTIN and Christophe OSSW ALD 19 Expert 2 Expert 1 Rock Cobb le Ripp le Sand Silt Shadow Othe r Rock - 1 2.87 2.7 2 4.42 3 .91 6 .41 0.22 Cobble 5.59 - 0.85 18 .44 3.85 0.0 4 0 Ripple 3 .12 3.38 - 30 .73 150.6 0 0.27 0.16 Sand 9.5 0 43.39 42. 60 - 5 24.33 0.51 0.57 Silt 6 .42 27.05 3 6.22 25 8.98 - 2.6 0 0.11 Shadow 3.82 0.15 2.13 1.3 8 0.50 - 0.41 Other 0 0.20 0.10 0.35 0 .31 0 .14 - T able 1: Matrix of conflict ( × 1 0 4 ) between the two experts. Hence, our application does not present a large conflict. W e have a pplied the con sensus ru le and the PCR5 rule with this model. The de - cision is giv en by the max imum of p ignistic probab ility . In most of the cases the decisions taken b y th e two r ules ar e the same. W e n ote a difference only on 0 .465 7% of the tiles. In deed, we are in the sev en classes case with o nly 0. 1209 of conflict, th e simulation given on th e figure 3 show that we have few ch ance that the d ecisions differ . 4 Conclusion In this paper we have proposed fi ve different models in o rder to take into acco unt two c lassical problems in unc ertain image classification (for tra ining or evaluation): the heter ogeneity o f the considered tiles and th e certainty o f the experts. These five models h av e b een developed in the DST and DSmT co ntexts. T he heter ogeneity of the tile an d th e certain ty o f th e exper t can be easily taken in to acc ount in the mo dels. Howe ver , if we w ant to have the plausibility of taking a decision on such a tile (with a conjunc tion A ∩ B ) the u sual decision function s (credibility , plausibility and pignistic probab ility) are not sufficient: they cannot allow a such decision. W e can take the decision on A ∩ B only if we consider the belief fu nction and if the mod el provides a belief on A ∩ B . W e ha ve also studied the decision acc ording to the con flict and to the comb ination rules: co njunctive consensus r ule a nd PCR5 ru le. The decision (taken w ith the m axi- mum of the credibility , the plausibility o r the pignistic probability) is the sam e in most of th e ca ses. For two experts, more classes le ads to more conflict an d to more cases giving a different decision with the dif ferent rules. W e h ave also illustrated one of the proposed mo dels o n real sonar images classified manually by two d ifferent experts. In this application the total conflict between the tw o experts is 0.1209 and we note a dif feren ce of decision only on 0.4 657% of the tiles. 20 HUMAN EXPERTS FUSION FOR IMA GE CLASSIFICATION W e can easily generalize our models for three or more experts and use the general- ized combination of the PCR5 gi ven by the equation (7). Of co urse t he conflict will be higher and the difference in the decision must be studied. Refer ences [1] M A RT I N , A . (2005) : Comp arative stud y of i nfo rmation fu sion methods for s onar images classification, The Eighth Internatio nal Confe rence o n Information Fu - sion, Philadelphia, USA, 25-29 July 2005. [2] X U L . , K R Z Y Z A K A . , S U E N C . Y . (1992): Metho ds of Combining Multiple Clas- sifiers and Their Application to Handwriting Recognition , IEEE T ransaction s on Systems, Man Cybernetics , V ol 22 (3), pp 418-435 . [3] L A M L . , S U E N , C . Y . (199 7): Application of Majority V oting to Pattern Recog- nition: An Analysis of Its Beha vior and Performan ce, I EEE T ransactio ns on Systems, Man, a nd Cybernetics - P art A: Systems an d Humans , V ol 27 (5), pp 553-5 68. [4] Z A D E H , L . (197 8): Fuzzy sets as a basis for a theor y o f po ssibility , Fuzzy Sets and Systems , V ol 1, pp 3- 28. [5] D U B O I S , D . , P R A D E , H . (1988 ): Possibility Th eory: An App roach to Comput- erized Processing of Uncertainty . Plenum Pr ess, New Y ork . [6] D E M P S T E R , A . P . (1 967): Upper and Lower pr obabilities induced by a mu lti val- ued mapping, Annals of Mathematical Statistics , V ol 83, pp 325- 339. [7] S H A F E R , G . (197 6): A mathema tical theor y of evidence, Princeton Un iversity Pr ess . [8] S M A R A N DA C H E , F . , D E Z E RT , J . ( E D I T O R S ) (2004) : App lications and Ad- vances of DSmT for Info rmation F usion , American Resear ch Pr ess Reho both . [9] S M E T S , P H . (1990) : Th e Combination of Ev idence in the T r ansferable Belief Model, IEEE T ransactions on P attern Analysis a nd Machine Intelligence , V ol 12(5) , pp 447- 458. [10] Y AG E R , R . R . (198 0): On th e Dempster-Shafer Fr amew ork and New Combina- tion Rules, Informations Sciences , V ol 41, pp 93-1 37. [11] D U B O I S , D . , P R A D E , H . (1988): Representatio n and Combination of u ncertainty with belief f unctions an d possibility m easures, Computa tional Intelligence , V ol 4, pp 244-2 64. Arnaud MARTIN and Christophe OSSW ALD 21 [12] S M E T S , P H . (19 93): Belief fun ctions: the Disjunctive Rule o f Com bination and the Gene ralized Bay esian Th eorem, Internatio nal Journal of Ap pr oximate Re a- soning , V ol 9, pp 1-3 5. [13] S M A R A N DAC H E , F. , D E Z E RT , J . (2005) : Informatio n Fusion Based on New Pr o- portional Confl ict Red istrib ution Ru les , In formatio n Fusion, Philade lphia, USA, 25-29 July 2005. [14] S M E T S , P H . (19 90): Constructing the p ignistic p robab ility function in a context of uncertain ty , Uncertainty in Artificial Intelligence , V ol 5, pp 29 -39. [15] D E Z E RT , J . , S M A R A N D AC H E , F . , D A N I E L , M . (2004) : The Gene ralized Pig- nistic T ransformation , I nform ation Fusion , Stockholm , Sweden , 28 Jun e-1 July 2004. ARNA UD MAR TIN is a teache r and researcher at the ENSIET A in the labor atory E 3 I 2 : EA387 6, Brest, France. He received a PhD degree in Signal Processing (2001), and Master in Pro bability ( 1998) . His resear ch interests are ma inly related to data fusion, data mining, signal processing especially for sonar and radar data. E -mail : Arnaud. Martin@ensieta.fr . CHRISTOPHE OSSW ALD is a teacher an d research er at teh E NSIET A in th e labo - ratory E 3 I 2 : EA3876 , Brest, Fran ce. He re ceiv ed a PhD d egree in Mathematics and Computer Science ( 2003) and engineer gr aduated from Ec ole Po lytechniqu e (pro mo- tion 199 4). His research inter ests are relate d to classification, data fusion and (h y- per)gr aphs theory . E-mail : Chr istophe.Osswald@ensieta.fr .