A Color Compensation Method Using Inverse Camera Response Function for Multi-exposure Image Fusion

A Color Compensation Method Using In v erse Camera Response Function for Multi-e xposure Image Fusion Artit V isav akitcharoen, Y uma Kinoshita and Hitoshi Kiya T okyo Metropolitan Uni versity , Hino, T okyo 191-0065, Japan Email: visav akitcharoen-artit@ed.tmu.ac.jp, kinoshita-yuma@ed.tmu.ac.jp, kiya@tmu.ac.jp Abstract —Multi-exposure image fusion is a method for pro- ducing an image with a wide dynamic range by fusing multiple images taken under various exposure values. In this paper , we discuss color distortion included in fused images, and propose a novel color compensation method for multi-exposure image fusion. In the proposed method, an inv erse camera response function (CRF) is estimated by using multi-exposure images, and then a high dynamic range (HDR) radiance map is recover ed. The color inf ormation of the radiance map is applied to images fused by con ventional multi-exposur e imaging to correct the color distortion. The proposed method can be applied to any existing fusion approaches for improving the quality of the fused images. Index T erms —Multi-exposure image, Image fusion, Color dis- tortion. I . I N T RO D U C T I O N The low dynamic range (LDR) of imaging sensors used in modern digital cameras is a major factor preventing cameras from capturing images as good as those with human vision. Accordingly , the interest of multi-exposure image fusion has recently been increasing. V arious research works on multi- exposure image fusion hav e so far been reported [1]–[4]. These fusion methods utilize a set of differently exposed images, i.e. multi-exposure images, and fuse them to produce an image with high quality . Ho wev er , con ventional multi- exposure image fusion methods ha ve not paid enough attention to the color of fused images, although they have paid attention to the spread of luminance. Because of such a situation, we pointed out that multi- exposure images have different colors, so the fused images hav e to include some color distortion [5]. T o improv e this issue, we focus on two insights: the constant hue plane in RGB color space [6] and inv erse camera response function (CRF). In this paper , an in verse CRF is estimated by using multi-exposure images, and then a high dynamic range (HDR) radiance map is recovered to estimate the correct colors of a scene. Next, the estimated color information is applied to a con ventional multi-exposure image fusion method on the constant hue plane in the RGB color space. The proposed method is not only a hue-preserving fusion method without gamut problem, but also a method applicable for any existing fusion methods to improv e the quality of fused images. Fig. 1: Proposed multi-exposure image fusion I I . P RO P O S E D M E T H O D A. Overview of Pr oposed Method The diagram of the proposed method is illustrated in Fig. 1. T o improv e the quality of images fused by using a con ven- tional image fusion method, we propose a color compensation method. Our approach is carried out as follo ws (See Fig. 1). 1) A fused image I f is produced by using a con ventional image fusion method. It can be expressed as I f = f ( I 1 , I 2 , ..., I N ) , (1) where f ( · ) is an image fusion function and I 1 , I 2 , ... I N are multi-exposure images. 2) An HDR radiance map is recovered from the multi- exposure images by estimating the inv erse camera re- sponse function (ICRF) [7] and then utilizing the esti- mated ICRF to reconstruct an HDR image I H . 3) Color compensation is carried out to improve the color distortion of the fused image I f . The constant hue plane in the RGB color space [6] is used for computing the maximally saturated color c f from I f , and c f is then replaced with a new one ˆ c f calculated from I H to obtain an improv ed image I 0 f . The follo wing is the detail of the proposed color compen- sation i.e. 2) and 3). B. Inver se Camera Response Function W e focus on the relationships between real scene luminance and pixel values [7]. Let a camera response function for mapping the scene radiance E i into a pixel v alue x ij be g ( · ) . Fig. 2: Constant hue plane in RGB color space The pixel value x ij at a spatial index i with an e xposure index j is written as x ij = g ( E i ∆ t j ) , (2) where ∆ t j is an exposure time for an exposure index j . In order to reproduce an HDR image from the multi- exposure images, Eq. (2) is solved to obtain the scene lu- minance map for each image by ln E i = g − 1 ( x ij ) − ln ∆ t j , (3) where g − 1 ( · ) is an in verse camera response function. The scene luminance maps are used to estimate an HDR image I H as y i = P N j =1 ω ( x ij )( g − 1 ( x ij ) − ln ∆ t j ) P N j =1 ω ( x ij ) , (4) where y i is referred to as pixel intensity in I H , and ω ( x ij ) is weight for each image. When the pixel value is closer to the middle of intensity range, which is given by ( x min + x max ) / 2 , the higher weight is used [7]. C. Constant Hue Plane in RGB Color Space W e utilize the constant hue plane in the RGB color space [6], as shown in Fig. 2. Each pix el of an input image is repre- sented in the RGB color space as x = ( a r , a g , a b ) , x ∈ [0 , 1] 3 . When a set of pixels has the same hue value, its intensity will align on the triangle, which consists of three vertices correspond to white, black and maximally saturated color represented by w = (1 , 1 , 1) , k = (0 , 0 , 0) and c = ( c r , c g , c b ) , respectiv ely . The maximally saturated color c is computed by c l = a l − min( x ) max( x ) − min( x ) . (5) where l ∈ { r , g , b } , and max( · ) and min( · ) are functions that return the maximum and minimum elements of the pixel x , respectiv ely . Note that the elements of c are in the range of [0 , 1] . When max( x ) = min( x ) , i.e. a r = a g = a b , the hue of the pixel x is not defined. The relationship between the RGB color space and the constant hue plane can be expressed by a linear combination with w , k , and c components as x = a r r + a g g + a b b , = a w w + a k k + a c c . (6) Let pixels at the same location in I f and I H be x f and x H , respectiv ely , namely , as x f = a wf w + a kf k + a cf c f , (7) x H = a wH w + a kH k + a cH c H . (8) The proposed color compensation method is carried out by replacing c f with c H , as ˆ x f = a wf w + a kf k + a cf c H . (9) By using this replacement, x f = a rf r + a g f g + a bf b in the RGB color space is modified, as ˆ x f = ˆ a rf r + ˆ a g f g + ˆ a bf b . (10) The constant-hue plane have been studied for impro ving some color distortions [5], [8]. I I I . E X P E R I M E N T In this experiment, 32 multi-exposure image sets were prepared form 32 HDR images I H 0 [9]. Each image set consists of 5 images with different e xposure vales, i.e. E V = [0 , ± 0 . 5 , ± 2] or E V = [0 , ± 1 , ± 2] . In addition, we ev aluated the color difference between I f and I H 0 in terms of the dif ference of hue v alues ∆ H ab between two images based on the CIEDE 2000 color -difference formula [10], which was published by the CIE [11]. T ABLE I: A verage values of ∆ H ab (with Mertens [1]) EV V alue Method ∆ H ab E V = [0 , ± 0 . 5 , ± 2] Con ventional 3.638 Proposed with c H 2.452 E V = [0 , ± 1 , ± 2] Con ventional 3.668 Proposed with c H 2.420 As shown in T able I, color distortions included in images generates by a con ventional fusion method were improved by our proposed method. I V . C O N C L U S I O N W e proposed a color compensation method for multi- exposure image fusion using HDR images reconstructed by estimating inv erse camera response functions. In an experi- ment, the proposed method was demonstrated to be effecti ve in terms of the CIEDE 2000 color -difference formula. R E F E R E N C E S [1] T . Mertens, J. Kautz, and F . V . Reeth, “Exposure fusion, ” in Proceedings of the 15th P acific Confer ence on Computer Graphics and Applications , IEEE, 2007, pp. 382–390. [2] M. Nejati, M. Karimi, S. R. Soroushmehr, N. Karimi, S. Sama vi, and K. Najarian, “Fast exposure fusion using exposedness function, ” in Pr oceedings of IEEE International Conference on Image Pr ocessing , IEEE, Sep. 2017, pp. 2234–2238. [3] Y . Kinoshita and H. Kiya, “Automatic exposure compensation using an image segmentation method for single-image-based multi-exposure fusion, ” APSIP A T ransactions on Signal and Information Pr ocessing , vol. 7, no. e22, Dec. 2018. [4] Y . Kinoshita and H. Kiya, “Scene Segmentation-Based Luminance Adjustment for Multi-Exposure Image Fusion, ” IEEE T ransactions on Image Pr ocessing , vol. 28, no. 8, pp. 4101–4116, Aug. 2019. [5] A. V isav akitcharoen, Y . Kinoshita, and H. Kiya, “Pure-color preserving multi-exposure image fusion, ” in International W orkshop on Advanced Image T echnology (IW AIT) 2019 , v ol. 11049, Jan. 2019, p. 110493X. [6] Y . Ueda, H. Misawa, T . Koga, N. Suetake, and E. Uchino, “Hue- preserving color contrast enhancement method without gamut problem by using histogram specification, ” in 2018 IEEE International Confer- ence on Ima ge Pr ocessing (ICIP) . IEEE, Oct. 2018, pp. 1128–1127. [7] P . E. Debe vec and J. Malik, “Reco vering high dynamic range radiance maps from photographs, ” in Pr oceedings of the 24th annual confer ence on Computer gr aphics and inter active techniques (SIGGRAPH’97) , Aug. 1997, pp. 369–378. [8] H. K obayashi and H. Kiya, “JPEG XT Image Compression with Hue Compensation for T wo-Layer HDR Coding, ” in Pr oceedings of IEEE International Conference on Consumer Electronics - Asia , Bangkok, Jun. 2019. [Online]. A vailable: http://arxi v .org/abs/1904.11315 [9] “High dynamic range image examples, ” http://www .anyhere.com/gward/ hdrenc/pages/originals.html. [10] G. Sharma, W . W u, and E. N. Dalal, “The CIEDE2000 color-dif ference formula: Implementation notes, supplementary test data, and mathemat- ical observations, ” Color Research & Application , vol. 30, no. 1, pp. 21–30, 2005. [11] ISO/CIE, “ISO/CIE 11664-6:2014 Colorimetry-P art 6: CIEDE2000 Colour-Dif ference Formula, ” 2014.