Non-linear aggregation of filters to improve image denoising

Non-linear aggregation of filters to impro v e image denoising Benjamin Guedj 1 and Juliette Rengot 2 1 Inria, F rance and Univ ersity College London, United Kingdom benjamin.guedj@inria.fr h ttps://bguedj.github.io 2 Ecole des P onts P arisT ec h, F rance juliette.rengot@eleves.enpc.fr Abstract. W e introduce a nov el aggregation metho d to efficiently per- form image denoising. Preliminary filters are aggregated in a non-linear fashion, using a new metric of pixel proximit y based on how the p o ol of filters reac hes a consensus. W e provide a theoretical bound to supp ort our aggregation sc heme, its n umerical performance is illustrated and w e sho w that the aggregate significan tly outp erforms each of the preliminary filters. Keyw ords: image denoising, statistical aggregation, ensem ble metho ds, collab orativ e filtering 1 In tro duction Denoising is a fundamental question in image pro cessing. It aims at improving the qualit y of an image by remo ving the parasitic information that randomly adds to the details of the scene. This noise may b e due to image capture condi- tions (lack of light, blurring, wrong tuning of field depth, . . . ) or to the camera itself (increase of sensor temp erature, data transmission error, appro ximations made during digitization, . . . ). Therefore, the challenge consists in remo ving the noise from the image while preserving its structure. Many metho ds of denoising already ha ve b een introduced in the past decades – while go o d p erformance has b een achiev ed, denoised images still tend to b e to o smo oth (some details are lost) and blurred (edges are less sharp). Seeking to improv e the p erformances of these algorithms is a v ery active research topic. The presen t pap er in tro duces a new approac h for denoising images, b y bring- ing to the computer vision communit y ideas developed in the statistical learning literature. The main idea is to com bine different classical denoising metho ds to obtain sev eral predictions of the pixel to denoise. As eac h classic metho d has pros and cons and is more or less efficient according to the kind of noise or to the image structure, an asset of our metho d is that is makes the b est out of each metho d’s strong p oin ts, p ointing out the ”wisdom of the cro wd”. W e adapt the strategy prop osed by the algorithm “COBRA - COmBined Regression Alternativ e” [ 2 , 10 ] 2 B. Guedj and J. Rengot to the sp ecific context of image denoising. This algorithm has b een implemented in the python library pycobra , av ailable on https://p ypi.org/pro ject/pycobra/ . Aggregation strategies may b e rephrased as collab orative filtering, since in- formation is filtered b y using a collaboration among m ultiple viewpoints. Collab- orativ e filters hav e already b een exploited in image denoising. [ 8 ] used them to create one of the most p erforming denoising algorithm: the blo ck-matc hing and 3D collab orative filtering (BM3D). It puts together similar patc hes (2D frag- men ts of the image) into 3D data arrays (called “groups”). It then produces a 3D estimate b y jointly filtering grouped image blo cks. The filtered blocks are placed again in their original p ositions, pro viding several estimations for each pixel. The information is aggregated to produce the final denoised image. This metho d is praised to w ell preserv e fine details. Moreo v er, [ 13 ] prov ed that the visual quality of denoised images can b e increased by adapting the denoising treatmen t to the lo cal structures. They prop osed an algorithm, based on BM3D, that uses different non-lo cal filtering mo dels in edge or smo oth regions. Collab- orativ e filters hav e also b een asso ciated to neural netw ork architectures, by [ 18 ], to create new denoising solutions. When sev eral denoising algorithms are a v ailable, finding the relev ant ag- gregation has been addressed b y sev eral works. [ 16 ] focused on the analysis of patc h-based denoising metho ds and shed light on their connection w ith statistical aggregation techniques. [ 6 ] prop osed a patch-based Wiener filter which exploits patc h redundancy . Their denoising approac h is designed for near-optimal p erfor- mance and reac hes high denoising quality . F urthermore, [ 17 ] sho wed that usual patc h-based denoising metho ds are less efficient on edge structures. The COBRA algorithm differs from the aforecited tec hniques, as it com- bines preliminary filters in a non-linear w ay . COBRA has been introduced and analysed b y [ 2 ]. The pap er is organised as follo ws. W e present our aggregation method, based on the COBRA algorithm in section 2 . W e then pro vide a thorough numerical exp erimen ts section ( section 3 ) to assess the p erformance of our metho d along with an automatic tuning pro cedure of preliminary filters as a byproduct. 2 The metho d W e now present an image denoising version of the COBRA algorithm [ 2 , 10 ]. F or each pixel p of the noisy image x , w e ma y call on M differen t estimators ( f 1 ...f M ). W e aggregate these estimators b y doing a weigh ted av erage on the in tensities : f ( p ) = P q ∈ x ω ( p, q ) x ( q ) P q ∈ x ω ( p, q ) , (1) Non-linear aggregation of filters to improv e image denoising 3 Fig. 1: General mo del and w e define the weigh ts as ω ( p, q ) = 1 M X k =1 1 ( | f k ( p ) − f k ( q ) | ≤  ) ≥ M α ! , (2) where  is a confidence parameter and α ∈ (0 , 1) a prop ortion parameter. Note that while f is linear with resp ect to the intensit y x , it is non-linear with resp ect to eac h of the preliminary estimators f 1 , . . . , f M . These weigh ts mean that, to denoise a pixel p , we av erage the intensities of pixels q such as a prop ortion at least α , of the preliminary estimators f 1 , . . . , f M ha ve the same v alue in p and in q , up to a confidence level  . Let us emphasize here that our procedure a verages the pixels’ intensities based on the weigh ts (which inv olve this consensus metric). The intensit y pre- dicted for each pixel p of the image is f ( p ) and the COBRA-denoised image is the collection of pixels { f ( p ) , p ∈ x } . This aggregation strategy is implemented in the python library pyc obr a [ 10 ]. The general scheme is presen ted in Figure 1 , and the pseudo-co de in Algo- rithm 1 . Users can control the n umber of used features thanks to the parameter “ patch siz e ”. F or eac h pixel p to denoise, we consider the image patch, centred on p , of size (2 · patch siz e + 1) × (2 · patch siz e + 1). In the exp eriments section, patch siz e = 2 is usually a satisfying v alue. Thus, for each pixel, we construct a v ector of nine features. The COBRA aggregation metho d has been introduced by [ 2 ] in a generic statistical learning framework, and is supported b y a sharp oracle b ound. F or the sak e of completeness, we repro duce here one of the key theorems. Theorem 1 (adapted from Theorem 2.1 in [ 2 ]). Assume we have M pr e- liminary denoising metho ds. L et | x | denote the total numb er of pixels in image x . L et  ∝ | x | − 1 M +2 . L et f ? denote the p erfe ctly denoise d image and b f denote the COBRA aggr e gate define d in ( 1 ) , then we have E h b f ( p ) − f ? ( p ) i 2 ≤ min m =1 ,...,M E [ f m ( p ) − f ? ( p )] 2 + C | x | − 2 M +2 , (3) 4 B. Guedj and J. Rengot Algorithm 1 Image denoising with COBRA aggregation INPUT: im noise = the noisy image to denoise p size = the pixel patch size to consider M = the num b er of COBRA machines to use OUTPUT: Y = the denoised image Xtrain ← training images with artificial noise Ytrain ← original training images (ground truth) cobra ← initial COBRA mo del cobra ← to adjust COBRA mo del parameters with respect to the data (Xtrain, Ytrain) cobra ← to load M COBRA machines cobra ← to aggregate the predictions Xtest ← feature extraction from im noise in a vector of size ( nb pixels, (2 · p size + 1) 2 ) Y ← prediction of Xtest by cobra Y ← to add im noise v alues lost at the b orders of the image, b ecause of the patch pro cessing, to Y wher e C is a c onstant and the exp e ctations ar e taken with r esp e ct to the pixels. What Theorem 1 tells us is that on av erage on all the image’s pixels, the quadratic error b etw een the COBRA denoised image and the p erfectly denoised image is upp er b ounded by the best ( i.e. , minimal) same error from the pre- liminary po ol of M denoising metho ds, up to a term whic h decays to zero as the num b er of pixels to the − 1 / M . As highlighted in the numerical exp erimen ts rep orted in the next section, M is of the order of 5-10 mac hines and this re- mainder term is therefore exp ected to b e small in most useful cases for COBRA. Note that in ( 3 ), the leading constan t (in front of the minimum) is 1: the oracle inequalit y is said to be sharp . Note also that con trary to more classical aggre- gation or model selection metho ds, COBRA mactches or outp erforms the b est preliminary filter’s p erformance, ev en though it do es not need to identify this c hampion filter. As a matter of fact, COBRA is adaptive to the p o ol of filters as the champion is not needed in ( 1 ). More comments on this result, and pro ofs are presen ted in [ 2 ]. 3 Numerical exp erimen ts This section illustrates the b ehaviour of COBRA. All co de material (in Python) to replicate the exp eriments presented in this pap er are av ailable at h ttps:// gith ub.com/bguedj/cobra denoising . Non-linear aggregation of filters to improv e image denoising 5 Fig. 2: The different kinds of noise used in our exp eriments. 3.1 Noise settings W e artificially add some disturbances to go o d qualit y images (i.e. without noise). W e fo cus on fiv e classical settings: the Gaussian noise, the salt-and-pepp er noise, the Poisson noise, the sp ec kle noise and the random suppression of patches (sum- marised in Figure 2 ). 3.2 Preliminary denoising algorithms W e fo cus on ten classical denoising metho ds: the Gaussian filter, the median filter, the bilateral filter, Chambolle’s metho d [ 5 ], non-lo cal means [ 3 , 4 ], the Ric hardson-Lucy deconv olution [ 14 , 15 ], the Lee filter [ 12 ], K-SVD [ 1 ], BM3D [ 8 ] and the inpainting metho d [ 7 , 9 ]. This wa y , w e in tend to capture differen t regimes of p erformance (Gaussian filters are known to yield blurry edges, the median filter is kno wn to b e efficient against salt-and-pepp er noise, the bilateral filter well preserv es the edges, non-lo cal means are praised to b etter preserve the details of the image, Lee filers are designed to address Syn thetic Ap erture Radar (SAR) image desp ec kling problems, K-SVD and BM3D are state-of-the-art approaches, inpain ting is designed to reconstruct lost part, etc.), as the COBRA aggregation sc heme is designed to blend together mac hines with v arious lev els of performance and adaptiv ely use the b est lo cal metho d. 3.3 Mo del training W e start with 25 images ( y 1 ...y 25 ), assumed not to b e noisy , that we use as “ground truth”. W e artificially add noise as describ ed ab ov e, yielding 125 noisy images ( x 1 ...x 125 ). Then tw o indep endent copies of each noisy image are created b y adding a normal noise: one go es to the data po ol to train the preliminary filters, the other one to the data p o ol to compute the weigh ts defined in ( 2 ) and p erform aggregation. This separation is intended to av oid ov er-fitting issues [as discussed in 2 ]. The whole dataset creation pro cess is illustrated in Figure 3 . 3.4 P arameters optimisation The meta-parameters for COBRA are α (ho w many preliminary filters m ust agree to retain the pixel) and  (the confidence level with which we declare tw o pixels iden tities similar). F or example, c ho osing α = 1 and  = 0 . 1 means that 6 B. Guedj and J. Rengot Fig. 3: Data set construction. w e imp ose that all the mac hines must agree on pixels whose predicted in tensities are at most differen t by a 0 . 1 margin. The python library pycobra ships with a dedicated class to derive the optimal v alues using cross-v alidation [ 10 ]. Optimal v alues are α = 4 / 7 and  = 0 . 2 in our setting. 3.5 Assessing the p erformance W e ev aluate the qualit y of the denoised image I d (whose mean is denoted µ d and standard deviation σ d ) with respect to the original image I o (whose mean is denoted µ o and standard deviation σ o ) with four differen t metrics. – Mean Absolute Error (MAE - the closer to zero the b etter) giv en by Σ N x =1 Σ M y =1 | I d ( x, y ) − I o ( x, y ) | N × M . – Ro ot Mean Square Error (RMSE - the closer to zero the b etter) given by r Σ N x =1 Σ M y =1 ( I d ( x, y ) − I o ( x, y )) 2 N × M . Non-linear aggregation of filters to improv e image denoising 7 – P eak Signal to Noise Ratio (PSNR - the larger the b etter) given by 10 · log 10  d 2 RMSE 2  with d the signal dynamic (maximal p ossible v alue for a pixel intensit y). – Univ ersal image Quality Index (UQI - the closer to one the b etter) given by cov ( I o , I d ) σ o · σ d | {z } ( i ) · 2 · µ o · µ d µ 2 o + µ 2 d | {z } ( ii ) · 2 · σ o · σ d σ 2 o + σ 2 d | {z } ( iii ) where term ( i ) is the correlation, ( ii ) is the mean luminance similarity , and ( iii ) is the con trast similarity [ 19 , Eq. 2]. 3.6 Results Our exp erimen ts run on the gray-scale “lena” reference image (range 0 - 255). In all tables, exp eriments hav e b een rep eated 100 times to compute descrip- tiv e statistics. The green line (resp ectively , red) identifies the b est (resp ectiv ely , w orst) p erformance. The yello w line identifies the best p erformance among the preliminary denoising algorithms if COBRA achiev es the b est p erformance. The first image is noisy , the second is what COBRA outputs, and the third is the difference b et ween the ideal image (with no noise) and the COBRA denoised image. R esults – Gaussian noise (Figur e 4 ). W e add to the reference image “lena” a Gaussian noise of mean µ = 127 . 5 and of standard deviation σ = 25 . 5. Un- surprisingly , the b est filter is the Gaussian filter, and the p erformance of the COBRA aggregate is tailing when the noise level is unknown. When the noise lev el is known, COBRA outp erforms all preliminary filters. Note that the bilat- eral filter gives b etter results than non-lo cal means. This is not surprising: [ 11 ] reac hes the same conclusion for high noise levels. R esults – salt-and-p epp er noise (Figur e 5 ). The prop ortion of white to blac k pixels is set to sp ratio = 0 . 2 and such that the prop ortion of pixels to replace is sp amount = 0 . 1. Even if the noise level is unknown, COBRA outp erforms all filters, ev en the champion BM3D. R esults – Poisson noise (Figur e 6 ). COBRA outp erforms all preliminary filters. R esults – sp e ckle noise (Figur e 7 ). When confronted with a sp eckle noise, CO- BRA outperforms all preliminary filters. Note that this is a difficult task and most filters hav e a hard time denoising the image. The message of aggregation is that even in adv ersarial situations, the aggregate (strictly) impro v es on the p erformance of the preliminary p o ol of metho ds. 8 B. Guedj and J. Rengot (a) Noisy image (b) COBRA (c) Diff. ideal-COBRA Fig. 4: Results – Gaussian noise. R esults – r andom p atches suppr ession (Figur e 8 ). W e randomly suppress 20 patc hes of size (4 × 4) pixels from the original image. These pixels become white. Unsurprisingly , the b est filter is the inpainting method – as a matter of fact this is the only filter which succeeds in denoising the image, as it is quite a sp ecific noise. R esults – images c ontaining sever al kinds of noise (Figur e 9 ). On all previous examples, COBRA matc hes or outp erforms the performance of the b est filter for eac h kind of noise (to the notable exception of missing patches, where inpain ting metho ds are sup erior). Finally , as the type of noise is usually unknown and even hard to infer from images, we are interested in putting all filters and COBRA to test when facing multiple t yp es of noise levels. W e apply a Gaussian noise in the upp er left-hand corner, a salt-and-p epp er noise in the upp er righ t-hand corner a noise of Poisson in the low er left-hand corner and a sp eckle noise in the low er righ t-hand corner. In addition, we randomly suppress small patchs on the whole image (see Figure 9a ). In this no w m uch more adv ersarial situation, none of the preliminary filters can ac hieve prop er denoising. This is the kind of setting where aggregation is the most interesting, as it will make the b est of each filter’s abilities. As a matter of fact, COBRA significan tly outp erforms all preliminary filters. Non-linear aggregation of filters to improv e image denoising 9 (a) Noisy image (b) COBRA (c) Diff. ideal-COBRA Fig. 5: Result – salt-and-p epp er noise. 3.7 Automatic tuning of filters Clearly , internal parameters for the classical preliminary filters may hav e a cru- cial impact. F or example, the median filter is particularly well suited for salt- and-p epper noise, although the filter size has to b e chosen carefully as it should gro w with the noise lev el (whic h is unknown in practice). A nice byproduct of our aggregated sc heme is that w e can also perform automatic and adaptive tuning of those parameters, by feeding COBRA with as many machines as p ossible v alues for these parameters. Let us illustrate this on a simple example: we train our mo del with only one classical method but with several v alues of the parameter to tune. F or example, we can define three mac hines applying median filters with differen t filter sizes : 3, 5 or 10. Whatever the noise lev el our approach achiev es the b est p erformance (Figure 10 ). This casts our approach onto the adaptiv e setting where we can efficien tly denoise an image regardless of its (unknown) noise lev el. 4 Conclusion W e hav e presented a generic aggregated denoising metho d—called COBRA— whic h improv es on the p erformance of preliminary filters, makes the most of their abilities (e.g., adaptation to a particular kind of noise) and automatically 10 B. Guedj and J. Rengot (a) Noisy image (b) COBRA (c) Diff. ideal-COBRA Fig. 6: Results – Poisson noise. adapts to the unkno wn noise lev el. COBRA is supp orted b y a sharp oracle inequalit y demonstrating its optimality , up to an explicit remainder term which quic kly go es to zero. Numerical exp eriment suggests that our metho d achiev es the b est p erformance when dealing with several t yp es of noise. Let us conclude b y stressing that our approac h is generic in the sense that any preliminary filters could b e aggregated, regardless of their nature and sp ecific abilities. Non-linear aggregation of filters to improv e image denoising 11 (a) Noisy image (b) COBRA (c) Diff. ideal-COBRA Fig. 7: Results – sp eckle noise. 12 B. Guedj and J. Rengot (a) Noisy image (b) COBRA (c) Diff. ideal-COBRA Fig. 8: Results – random suppression of patches. Non-linear aggregation of filters to improv e image denoising 13 (a) Noisy image (b) COBRA (Un- kno wn noise) (c) COBRA (Kno wn noise) (d) Bilateral fil- ter (e) Non-lo cal means (f ) Ric hardson- Lucy decon volu- tion (g) Gaussian fil- ter (h) Median filter (i) TV Cham- b olle (j) Inpain ting (k) K-SVD (l) BM3D (m) Lee filter Fig. 9: Denoising an image afflicted with multiple noises types. 14 B. Guedj and J. Rengot Fig. 10: Automatic tuning of the median filter using COBRA. Bibliograph y [1] Aharon, M., Elad, M., Bruckstein, A., et al.: K-svd: An algorithm for design- ing ov ercomplete dictionaries for sparse representation. IEEE T ransactions on signal pro cessing 54 (11) (2006) 4311 [2] Biau, G., Fisc her, A., Guedj, B., Malley , J.D.: Cobra: A com bined regression strategy . Journal of Multiv ariate Analysis 146 (2016) 18 – 28 [3] Buades, A., Coll, B., Morel, J..: A non-lo cal algorithm for image denois- ing. In: 2005 IEEE Computer So ciety Conference on Computer Vision and P attern Recognition (CVPR’05). V olume 2. (2005) 60–65 vol. 2 [4] Buades, A., Coll, B., Morel, J.M.: Non-lo cal means denoising. Image Pro- cessing On Line 1 (2011) 208–212 [5] Cham b olle, A.: T otal v ariation minimization and a class of binary mrf mo dels. Energy Minimization Metho ds in Computer Vision and Pattern Recognition 3757 (2005) 132–152 [6] Chatterjee, P ., Milanfar, P .: P atch-based near-optimal image denoising. IEEE T ransactions on Image Pro cessing 21 (4) (2012) 1635–1649 [7] Ch uiab, C., Mhask ar, H.: Mra contextual-reco v ery extension of smo oth functions on manifolds. Applied and Computational Harmonic Analysis 28 (01 2010) 104–113 [8] Dab o v, K., F oi, A., Katko vnik, V., Egiazarian, K.: Image denoising b y sparse 3-d transform-domain collab orative filtering. IEEE T ransactions on image pro cessing 16 (8) (2007) 2080–2095 [9] Damelin, S., Hoang, N.: On surface completion and image inpainting by biharmonic functions: Numerical asp ects. In ternational Journal of Mathe- matics and Mathematical Sciences 2018 (01 2018) 8 [10] Guedj, B., Sriniv asa Desik an, B.: Pycobra: A python to olb ox for ensem ble learning and visualisation. Journal of Machine Learning Researc h 18 (190) (2018) 1–5 [11] Kumar, B.S.: Image denoising based on non-lo cal means filter and its metho d noise thresholding. Signal, image and video pro cessing 7 (6) (2013) 1211–1227 [12] Lee, J.S., Jurk evich, L., Dew aele, P ., W am bacq, P ., Oosterlinck, A.: Sp eckle filtering of syn thetic ap erture radar images: A review. Remote sensing reviews 8 (4) (1994) 313–340 [13] Liu, J., Liu, R., Chen, J., Y ang, Y., Ma, D.: Collab orativ e filtering de- noising algorithm based on the nonlo cal centralized sparse representation mo del. In: 2017 10th International Congress on Image and Signal Pro cess- ing, BioMedical Engineering and Informatics (CISP-BMEI). (2017) [14] Lucy , L.: An iterative technique for the rectification of observed distribu- tions. Astronomical Journal 19 (06 1974) 745 [15] Ric hardson, W.H.: Ba yesian-based iterative metho d of image restoration. Journal of the Optical So ciet y of America 62 (1972) 55–59 16 B. Guedj and J. Rengot [16] Salmon, J., Le Pennec, E.: Nl-means and aggregation pro cedures. In: 2009 16th IEEE In ternational Conference on Image Pro cessing (ICIP). (Nov 2009) 2977–2980 [17] Salmon, J.: Agr ´ egation d’estimateurs et m´ etho des ` a patc h p our le d ´ ebruitage d’images num ´ eriques. PhD thesis, Univ ersit´ e P aris-Diderot-Paris VI I (2010) [18] Strub, F., Mary , J.: Collab orative filtering with stack ed denoising auto en- co ders and sparse inputs. I n: NIPS w orkshop on machine learning for eCom- merce. (2015) [19] W ang, Z., Bovik, A.C.: A universal image qualit y index. IEEE signal pro cessing letters 9 (3) (2002) 81–84