No reference image quality assessment metric based on regional mutual information among images

No reference image qualit y assessmen t metric based on regiona l m utual information among images Vina y Kumar ∗ Viv ek Singh Ba w a † Rah ul Upadh y a y ∗ Abstract With the inclusion o f camera in dail y life, an automatic no ref erence image qualit y ev aluation index is re quired for automatic c lassification of images. The present man uscripts proposes a n ew No Reference Regional Mutual Information based technique for ev aluating the q ualit y of an im- age. W e use regional mutual information on sub sets of the complete image. Proposed tec hniqu e is tested o n four b enchmark natural image databases, and one b enchmark synthetic database. A comparativ e analysis with clas- sical and state-of-art metho ds indicate sup eriorit y of the present technique for h igh q u alit y images and comparable for other images of the resp ective databases. 1 In tro duction With the adven t of inexp ensive and go o d quality mobile ca meras stor age, trans- mission and co mpression of images has b ecome a standar d practice a mong tech- nical and non technical masses. Large n umber of peo ple have mobile phones with camera capturing trillions of photogra phs every year [1 ], approximately 24 billion s e lfies were uploaded to Go ogle in year 2015 [2] and increa s ing exp onen- tially with every pa ssing year. Unlimited space for uploading ima g es on Go o gle photos (and large space on o ther w eb servers; for example, Flickr, Pin terest, etc) facilitates and influences p eople to capture many photogra phs o f the same situation. Searching the go o d qua lity images fro m this ever (exp onentially) increasing large qua nt ity is imp os s ible tas k for a human b eing. Therefore , it b e- comes per tinent to design and develop b etter a utomatic a nd no-reference image quality ass essment system. These systems will help; for example, in ev aluating the image informa tion a nd (p oss ibly) r etain the b est out of plethora , find o ut the quality in real time, se le c ting c a mera settings for best results, etc. This drives r e s earchers to dev elop b etter auto no reference image quality mea sure- men t techniques [13–15, 18, 19, 29, 30]. ∗ Departmen t of E l ectronics & Comm unication Engineer ing, Thapar Institute of Engineer- ing and T echnology , Patiala † Visual Artificial Int elligence Laboratory , Oxford Brookes U nive rsity , UK. 1 Researchers gener ally talk ab out three types of image qualit y asse s sment (IQA) techniques: 1. full reference [3, 7, 11, 27, 28, 3 5, 36, 38], 2. reduced r e fer ence IQA [4, 2 6], and 3. no-re ference IQA [6, 8, 1 6, 37]. First t ype o f IQ A as sumes that human b eings are sensitive to degradations, second indica tes that we are more sensitive to few key feature s e x tracted fro m the image. The current pro po sed tec hnique lies in the la st catego ry . V ario us tec hniques for o b jective image quality measurement a re discussed in literature. Since h uman visual system is a complex set of de cision ma king pro cesses, av ailable IQA metho ds are still not as go od a s the human visual decisions. W e discuss s o me of the relev a nt a nd prev ailing metho ds in rest of the present section. W u et al [34] used measur ement o f blo cking effect in ho rizontal and vertical directions a nd differences at blo ck bo undaries in horizontal a nd vertical direc- tions, resp ectively . T an et al [25] analyzes magnitude and phase information in a harmonics to measure the qua lit y of the ima ge. Another [24] model w as developed for measuring blo ck effects in a n ima g e. W a ng et al [31] used ener g y based measure ments to find the blo cking ar tifacts in an image. These blo cking effects beco me fundamen tal building blocks for measurement of quality o f a n image. While tra ns mitting or sto r ing, imag e quality (IQ) measurement pla ys a cru- cial ro le to ev aluate and cho o se the correct imag e. The ultimate goal o f IQ mea- surement is assigning a quantitativ e v alue to p erception to human observers. Researchers p erfor m this task with the help of crowd s ourcing and acquiring Mean Opinion Scor e (MOS). MOS or its mo dified versions compared with the IQA v alues beco me basis for quality o f the IQA index. In the next section we discuss prop osed metho d follow ed, in sectio n 3, by exp erimental results a nd discussion. W e clo se the manuscript with co nclusion and references. 2 Metho dology The prop o sed index No Reference Reg ional Mutual Informatio n (NrMI) pr edicts the quality of an image with the help of following pr o cedure. Given an image matrix Φ( x, y ) ∈ Z n × m . Another version of ma trix Φ( x, y ) is created a nd is depicted by Φ ′ θ ( x ′ , y ′ ) , where  x ′ y ′  =  cosθ − sinθ sinθ cosθ   x y  (1) T o ma ke Φ and Φ ′ θ of sa me size equations 2 and 3 are applied. Φ vec ( v ) = ve c (Φ ′ θ ( x ′ , y ′ )) (2) 2 Φ θ v ( x, y ) = v e c − 1 (Φ vec ( v )) : Φ θ ( x, y ) ∈ R ab → R n × m (3) W e divide Φ( x, y ) into disjoint gro up o f n sub-matrices, η a × b k : k ∈ Z > where η k : η k ⊂ Φ . η k contains q mem ber of p erfect subsets o f Φ (such that k / m ∈ Z > ), for it is obvious that if a subset is p erfect, then there is no information loss. Ev ery element of η k represents a segment of original image Φ . Φ ′ θ ( x ′ , y ′ ) is divided into s ub-matrices η a × b k,θ ( ≡ η k ). W e choose size of η k (and conse q uen tly η k,θ ) to be 3 × 3 , which makes sure that the v alues within η k will not b e v arying significantly except when sub- matrix lies at an edg e in Φ( x, y ) (or Φ ′ θ ( x ′ , y ′ ) ). The v alue of θ is π / 2 , one can cho ose any v alue fo r θ but π / 2 provides max im um s hift, a nd equation 1 is rewritten as  x ′ y ′  =  − y x  (4) which r elates image matrices Φ and Φ ′ θ b y equa tio n 5 Φ( x, y ) = Φ ′ θ ( − y , x ) (5) Let sub-matr ic e s η k (and η k,θ ) b e r epresented by η k =   e 11 e 12 e 13 e 21 e 22 e 23 e 31 e 32 e 33   η k,θ =   e 11 ,θ e 12 ,θ e 13 ,θ e 21 ,θ e 22 ,θ e 23 ,θ e 31 ,θ e 32 ,θ e 33 ,θ   (6) F rom ma trices of equation 6 ca lculate matrix M e M e = [ e 11 e 12 e 13 e 11 ,θ e 12 ,θ e 13 ,θ e 21 . . . e 23 e 21 ,θ . . . e 33 ,θ ] (7) Cent er the v alues a t the origin and r epresent it by M e, 0 b y M e, 0 = M e − 1 N N X i p i (8) where p i are elements of matrix M e and N = 9 + 9 = 18 . Find cov ariance C = 1 N M e, 0 M T e, 0 (9) Estimate joint en tropy 1 H g ( C ) 1 Join t and marginal entro py is giv en b y [22] H g (Σ d ) = l og ((2 π e ) d 2 det (Σ d ) 1 2 ) (10) whic h represen ts the en trop y of a normally distributed set of p oint s in ℜ d with cov ariance matrix Σ d . 3 Estimate mar ginal en tropy1 H g ( C A ) and H g ( C B ) , where C A and C B are top left a nd b ottom rig ht d 2 × d 2 matrices of C . d is a relationship defined as d = 2(2 r + 1) 2 (11) where, r is the size of sub matrix under considera tion; that is, the size of MB for whic h we a re going to calculate the similarity within the matrix; for example in Fig ure v alue of r is 1 . Calculate Regiona l Mutual Information M r mi = H g ( C A ) + H g ( C B ) − H g ( C ) (12) M r mi gives a mea sure of regional m utual information b etw een Φ( x, y ) a nd Φ θ ( x ′ , y ′ ) . A weigh t function for RMI is calculated with equa tion of Φ vec ( v ) is ca lculated next Φ wg = E  (Φ vec ( v ) − E [Φ vec ]) 2  (13) The rela tive quality of an image is given by N r M I i = M r mi,i ∗ Φ wg ,i (14) where i ∈ i th image in the image sequence. 3 Exp eriment al Results In this section w e v a lidate o ur metho d through application o n v arious b ench- mark state-of-ar t and classical databas es. Exper iment s ar e conducted with five standard databa ses of natural and one of synthetic ima g es. The natural image databases are TID 2008 [21] with 1699 images, TID 2013 [20] with 2483 images, CID 2013 [10], LIVE [23], MEFD with 550 imag es each. While ESPL [9], a database consisting of 55 0 synt hetic images , is used for ev aluation of the cur - rent algo rithm. F or ob jective ev aluation SR CC (Spear man’s Rank Co rrelation Co efficient) and PLCC (Pearson Linear Cor r elation Co efficient) matrices a re used. These metrics give a measure of prediction monotonicity a nd linearity , resp ectively . T able 1 presents a c o mparative view of v a rious index of quantitativ e quality measures. Blue color v alues in table 1 indicate b est results. Since no-re fer ence quality measurement system requir es complex set of interdependent parameters to work a s efficiently as human b eings, therefore every system has certain ad- v antage ov er others under certain co nditions. F rom the table it b ecomes clear that prop osed method ev aluates the images b etter than SSIM for most of the databases. Since the prop ose d metho d uses underlying reg ional geo metric infor- mation by splitting the set in to disjoint g roup o f sub-s ets; therefor e every small change in geometry (including pr esence of undetectable nois e for h uman v isual system) changes the qualitative measur e. 4 Database Statistical Measureme nt SSIM [32] NR [33] NJQA [5] NR [12] MUG NR [17] MUG + NR [17] PM TID 2008 [21] PLCC 0.954 0.952 0.944 0.951 0.941 0.953 0.868 SRC C 0.925 0.913 0.8993 0.917 0.917 0.924 0.832 TID 2013 [20] PLCC 0.954 0.953 0.948 0.955 0.942 0.955 0.887 SRC C 0.9200 0.927 0.886 0.931 0.908 0.919 0.842 CID 2013 [10] PLCC 0.979 0.975 0.954 0.979 0.9679 0. 972 0.789 SRC C 0.955 0.955 0.925 0.957 0.930 0.937 0.798 LIVE [23] PLCC 0.979 0.979 0.956 0.976 0.965 0.973 0.962 SRC C 0.946 0.974 0.956 0.973 0.959 0.968 0.959 ESPL [9] PLCC 0.943 0.960 0.809 0.962 0.940 0.937 0.962 SRC C 0.904 0.933 0.739 0.933 0.928 0.927 0.959 T able 1: Performance co mparison of no reference image qualit y measure for TID 200 8 [21], TID 2 013 [20], CID 20 13 [10], LIVE [23], MEFD, and ESPL [9] databases. Blue co lor v alues represent best p erfor ming technique in ter ms of corresp o nding SRCC and PLCC statistical mea sures. Specifica lly with images o f high qua lity (databases MEFD a nd ESPL) the prop osed metho d p erforms muc h b etter than SSIM [3 2] a nd other state-o f-a rt tech niques. Since prop osed techni que considers underlying g eometry o f the im- age, high q ua lit y images disto rted b y small amo un t of noise pro duce low er v alue of the quality index. This low er v alue in turn will b e helpful to take cor r ective measures to develop noise remov al or b etter compressio n a lg orithms. 4 Conclusion Present manuscript inv e s tigates the pro blem o f no-refere n ce quality as sessment. A novel technique has bee n pr o p osed for the assessment based on underlying geometry of the ima g e. The technique is applied on v ario us databases with different types o f images. Re s ults show interesting trend and promising p erfor- mance when compared with ex isting litera ture. Since metho d utilizes mutual information appro ach it was able to render b etter results fo r high quality imag es. In future we aim to study the effects of current techni que b y calculating RMI on weigh ted ima g e segments. The weights will b e calculated ba s ed o n the impo r tance of the region in the image s, which in turn dep ends on p oint of focus in human vis ual system. References [1] How Ma ny Photos Will b e T aken in 2020? [2] 28+ Selfie Statistics, Demogr aphics, & F un F acts (2022 ), Mar ch 202 2. 5 [3] Y uming F ang, Kai Zeng, Zhou W ang, W eisi Lin, Zhijun F ang, and Chia- W en Lin. Ob jective q uality assessment for image r etargeting based o n structural similarity . IEEE Journal on Emer ging and Sele cte d T opics in Cir cuits and Systems , 4(1):95 – 105, 2 0 14. [4] Xinbo Ga o, W en Lu, Dacheng T a o, and Xuelong Li. Image quality a ssess- men t based on m ultiscale geometric analysis. IEEE T r ansactions on Image Pr o c essing , 18(7):14 09–1 423, 2009. [5] S Alireza Golestaneh and Damon M Chandler. No-r e fer ence quality a ssess- men t of jpeg images via a quality relev ance map. IEEE signal pr o c essing letters , 21 (2):155– 158, 2014 . [6] Ke Gu, Shiqi W ang, Guangtao Zhai, Siwei Ma, Xiaok ang Y ang, W eisi L in, W enjun Zhang, and W en Gao. Blind quality assessment of tone-mapp e d images via a nalysis of infor ma tion, natura lness, and s tructure. IEEE T ra ns- actions on Multime dia , 18(3):43 2–44 3, 2016. [7] Ke Gu, Gua ngtao Z hai, Xia ok ang Y ang, and W enjun Z hang. An e fficient color imag e quality metric with lo c a l-tuned-global mo del. In Image Pr o- c essing (ICIP), 2014 IEEE International Confer enc e on , pages 506 – 510. IEEE, 2 014. [8] Ke Gu, Guang tao Zhai, Xiaok a ng Y ang, and W enjun Zhang. Using free energy principle for blind ima ge qualit y assess ment . IEEE T r ansactions on Multime dia , 17(1):50–6 3, 2 0 15. [9] Debarati K undu and Brian L Ev ans. F ull-reference visua l quality as sess- men t for synthetic images: A sub jective study . In Image Pr o c essing (ICIP), 2015 IEEE International Confer enc e on , pages 2374 – 2378 . IEEE, 2015 . [10] Eric C Larson and Damon M Chandler. Most apparent distortion: full- reference image quality a ssessment and the role of stra tegy . Journal of Ele ctr onic Imaging , 19 (1):01100 6–01 1006, 2 010. [11] Leida Li, W enha n Xia, Y uming F ang, Ke Gu, Jinjian W u, W eisi Lin, and Jiansheng Qian. Color image quality assessment based on spar se repr e s en- tation and reconstr uctio n residual. Journal of Visual Communic ation and Image R epr esentation , 38 :550–5 60, 2016. [12] Leida Li, Y u Zhou, Jinjian W u, W eisi Lin, and Haolia ng Li. Gridsar : Grid s trength and reg ula rity for ro bus t ev aluation of blo cking a rtifacts in jpeg images. Journal of Visual Co mmunic ation and Image R epr esentation , 30:153 –163 , 2015. [13] Luhong Liang, Shiqi W ang , Jianhua Chen, Siwei Ma , Debin Zhao , and W en Gao. No-reference p erceptual image quality metric using g radient profiles for jp eg20 00. S ignal Pr o c essing: Image Commun ic ation , 25 (7):502– 516, 2010. 6 [14] W eisi Lin and C-C Jay Kuo. Perceptual visua l qua lit y metrics: A survey . Journal of Visual Communic ation and Image R epr esentation , 22(4):29 7– 312, 2 011. [15] Anmin Liu, W eisi Lin, and Manish Narwaria. Image quality a s sessment based o n gra dient similarity . IEEE T r ansactions on Image Pr o c essing , 21(4):150 0–15 1 2, 2012. [16] Anish Mittal, An ush Kris hna Mo orthy , and Alan Co nrad Bovik. No- reference image qua lit y assessment in the spatial domain. IEEE T r ans- actions on Image Pr o c essing , 21(12):4 6 95–4 708, 2012. [17] Hossein Ziaei Nafchi, Atena Shahkolaei, Rachid Hedjam, a nd Mohamed Cheriet. Mug: A parameter les s no-reference jpeg quality ev aluator r o- bust to blo ck size and misa lignment. IEEE Signal Pr o c essing L etters , 23(11):15 77–1 5 81, 2016. [18] Manish Narwaria a nd W eisi Lin. Ob jective image qualit y assessment ba sed on supp or t vector regress ion. IEEE T ra nsactions on Neur al Networks , 21(3):515 –519 , 2 010. [19] Manish Narwaria, W eisi Lin, Ian Vince McLoughlin, Sa bu Emmanuel, and Liang-Tien Chia. F ourier transform-ba sed sca la ble image quality measure. IEEE T r ansactions on Image Pr o c essing , 2 1 (8):3364 –3377 , 20 1 2. [20] Nikola y Ponomarenko, O leg Ier emeiev, Vladimir Lukin, Karen Egiazar- ian, Lina Jin, Jaakko Astola , Beno it V ozel, Kacem Chehdi, Marco Car li, F ederica Battisti, et a l. Colo r image database tid201 3: P eculiarities a nd preliminary results. In Visual Information Pr o c essing (EUVIP), 2013 4th Eur op e an W orkshop on , pa ges 10 6–111 . IEEE, 201 3. [21] Nikola y P onomar enko, Vladimir Lukin, Alexander Zelensky , Karen Egiazarian, Marco Carli, and F ederica Battisti. Tid2008 -a database for ev aluation of full-refere nce v is ual qua lity assessment metric s . A dvanc es of Mo dern Ra dio ele ctr onics , 10(4):30 –45, 2 0 09. [22] F azlo llah M Reza. An intr o duction to information the ory . Inc, New Y ork, 1994. [23] Hamid R Sheikh, Muhammad F Sabir, a nd Alan C Bovik. A statistical ev al- uation of recent full reference image quality asses sment a lgorithms. IEEE T r ansactions on image pr o c essing , 15 (11):3440 –345 1, 2006 . [24] KT T an a nd Mohammed Ghanbari. Blo ckiness detection for mpeg 2-co ded video. IEEE Signal Pr o c essing L etters , 7(8):21 3–215 , 2000. [25] KT T an and Mohammed Ghan bari. F requency domain measur ement of blo ckiness in mpe g -2 co ded v ideo. In Image Pr o c essing, 2 000. Pr o c e e dings. 2000 International Confer enc e on , volume 3, pages 977–9 80. IEEE, 2 000. 7 [26] Dacheng T ao, Xuelong Li, W en Lu, and Xinbo Gao . Reduced- reference iqa in cont ourlet domain. IEEE T r ansactions on Syst ems, Man, and Cyb er- netics, Part B (Cyb ernet ics) , 39(6):16 23–16 27, 2009. [27] Shiqi W ang, Abdul Rehman, Zhou W ang , Siwei Ma, a nd W en Ga o. Ssim- inspired divisive norma liza tion for p erceptual video co ding . In Image Pr o- c essing (ICIP), 2011 18 th IEEE International Confer enc e on , pa ges 1 657– 1660. IEEE, 2 011. [28] Shiqi W ang, Abdul Rehman, Zhou W ang , Siwei Ma, a nd W en Ga o. Ssim- motiv ated ra te- distortion optimization for video co ding. IEEE T r ansactions on Cir cuits and Syst ems for Vide o T e chnolo gy , 2 2 (4):516– 529, 2012 . [29] Shiqi W ang, Xiang Zha ng , Siwei Ma, and W en Gao. Reduced reference image quality assessment using entrop y of primitiv es. In Pictur e Co ding Symp osium (PCS), 2013 , pages 1 93–1 96. IEEE, 20 13. [30] Z W ang and Alan C Bovik. Image a nd m ultidimensional signal pr o cessing-a univ ersa l image qua lity index. IEEE Signal Pr o c essing L et ters , 9(3):81 –84, 2002. [31] Zhou W ang, Alan C Bovik, a nd BL Ev an. Blind mea surement of blo cking artifacts in images. In Image Pr o c essing, 2000. Pr o c e e dings. 200 0 Int erna- tional Confer enc e on , volume 3, pages 981–9 84. Ieee, 20 00. [32] Zhou W ang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Ima ge quality assessment: from err or v is ibilit y to structural similarity . IEEE tr ansactions on image pr o c essing , 1 3(4):600 –612, 200 4. [33] Zhou W a ng, Hamid R Sheikh, and Alan C Bovik. No-r eference p erceptual quality ass essment of jp eg compressed images. In Image Pr o c essing. 2002. Pr o c e e dings. 200 2 International Confer enc e on , v olume 1, pag e s I–I. IEEE, 2002. [34] HR W u and M Y uen. A g eneralized blo ck-edge impair ment me tric for video co ding. IEEE Signal Pr o c essing L etters , 4(11):3 17–32 0, 1997. [35] Jinjian W u, W eisi Lin, Guangming Shi, and Anmin Liu. Perceptual q ua lit y metric with internal genera tive mechanism. IEEE T r ansactions on Image Pr o c essing , 22(1):43 –54, 20 13. [36] Lin Zhang, Ying Shen, and Ho ngyu Li. V si: A v is ual sa liency-induced index for per ceptual image quality as sessment. IEEE T r ansactions on Image Pr o c essing , 23(10 ):4270– 4281 , 2 014. [37] Lin Zhang, Lei Zhang, and Alan C Bo vik. A featur e - enriched completely blind imag e quality ev aluator. IEEE T ra nsactions on Image Pr o c essing , 24(8):257 9–25 9 1, 2015. 8 [38] Lin Z hang, Lei Zha ng , Xuanqin Mou, and Da vid Zha ng. F sim: A feature similarity index for image qua lity a ssessment. IEEE tr ansactions on Image Pr o c essing , 20(8):23 78–2 386, 2011. 9 This figure "PLCC.jpg" is available in "jpg" format from: This figure "PLCC.png" is available in "png" format from: This figure "SRCC.jpg" is available in "jpg" format from: This figure "SRCC.png" is available in "png" format from: